From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including Deepseek-V3.2. Our code is publicly available.

Large language model agents are increasingly used in settings where they interact with the world over time, such as games, coding tasks, or tool-use tasks. A key challenge is that these agents often repeat mistakes because they do not reliably turn past experience into useful guidance for future decisions. Existing memory systems usually rely on hand-written rules or prompts to summarize what happened, but these summaries may miss important patterns or give advice that the agent cannot easily act on. We introduce MemoPilot, a trainable memory assistant that learns how to update an agent’s memory after each interaction. Instead of changing the main language model, MemoPilot keeps the agent fixed and learns to write better memories that help the agent improve over a sequence of tasks. We train this memory assistant using feedback from the agent’s later performance, so the memory is optimized for whether it actually helps, not just whether it looks reasonable. In strategic games, MemoPilot helps agents adapt faster and perform better than using no memory, raw history, or prompt-based memory updates. This suggests that learning how to remember can be an effective way to make AI agents improve from experience without retraining the whole model.