CoMem: Context Management with A Decoupled Long-Context Model

Context management enables agentic models to solve long-horizon tasks through iterative summarization of previous interaction histories. However, this process typically incurs substantial decoding overhead for the extra summarization tokens, which significantly affect the end-to-end response latency at deployment. In this paper, we introduce CoMem, a novel framework that decouples memory management from the primary agent workflow, enabling these processes to execute in parallel. We propose a $k$-step-off asynchronous pipeline that overlaps the memory model's summarization with the agent's inference, effectively masking the latency of context processing. To ensure robustness under this asynchronous setting, we introduce a reward-driven training strategy that aligns the memory model to capture sufficient statistics for the agent's decision-making. Theoretical analysis confirms that CoMem offers a superior efficiency-effectiveness trade-off compared to coupled architectures. Our extensive experimental results on SWE-Bench-Verified show that CoMem provides 1.4x latency improvements upon vanilla long-context solutions while preserving most of the performance. Furthermore, we demonstrate that these latency gains scale favorably with increased system throughput, offering a modular path forward for the independent optimization of agent reasoning and memory compression.

LLM agents that complete complex tasks, like fixing software bugs or navigating websites, often need to remember long histories of their past actions. However, as these histories grow, the agent slows down significantly because it must re-read everything before deciding what to do next. We propose CoMem, a framework that gives LLM agents a dedicated "memory assistant" — a smaller, faster model whose only job is to continuously summarize what has happened so far. Instead of making the main AI read the full history, we run the memory assistant in parallel, so the main AI receives a concise summary without waiting. We also developed a training method that teaches the memory assistant what information is actually important for decision-making. In experiments on software engineering and web tasks, CoMem reduces response time by 1.4x while maintaining task performance. These efficiency gains grow further as the system handles more requests simultaneously, making capable LLM agents more practical to deploy at scale.