Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs

Most evaluations of External Memory Module assume a static setting: memory is built offline and queried at a fixed state. In practice, memory is streaming: new facts arrive continuously, insertions interleave with retrievals, and the memory state evolves while the model is serving queries. In this regime, accuracy and cost are governed by the full memory lifecycle, which encompasses the ingestion, maintenance, retrieval, and integration of information into generation. We present Neuromem, a scalable testbed that benchmarks External Memory Module under an interleaved insertion-and-retrieval protocol and decomposes its lifecycle into five dimensions including memory data structure, normalization strategy, consolidation policy, query formulation strategy, and context integration mechanism. Using three representative datasets LoCoMo, LONGMEMEVAL, and MemAgentBench, Neuromem evaluates interchangeable variants within a shared serving stack, reporting token-level F1 and insertion/retrieval latency.Overall, we observe that performance typically degrades as memory grows across rounds, and time-related queries remain the most challenging category. The memory data structure largely determines the attainable quality frontier, while aggressive compression and generative integration mechanisms mostly shift cost between insertion and retrieval with limited accuracy gain.

Large language models are increasingly used in long-running conversations and interactive applications, where they need to remember past information and use it later. However, existing evaluations often test memory systems only after all information has already been stored, which does not reflect real use cases where new information arrives continuously while the system is also being queried. This paper introduces Neuromem, a testbed for evaluating external memory systems under this more realistic streaming setting. Neuromem breaks the memory process into several stages, such as how information is stored, updated, retrieved, and added back into the model’s context. This makes it possible to understand which design choices improve answer quality and which mainly add computational cost. Across several long-memory benchmarks, we find that the structure used to store information strongly affects performance, while expensive generation-based memory operations often add latency with limited benefit under real-time constraints. These results provide practical guidance for building more efficient and reliable memory systems for long-horizon AI applications.