IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies.

Large language models are increasingly used to read and reason over very long inputs, such as long documents, codebases, or conversations. However, remembering all earlier information during generation requires a large amount of computer memory, which makes long-context use expensive and difficult to deploy. This paper introduces IndexMem, a method that helps a language model use its memory more efficiently. Instead of keeping everything, the model learns to identify which earlier pieces of information are most likely to be useful later and keeps those parts. At the same time, the information that is removed is not simply thrown away. IndexMem stores a compact summary of the removed information in a small memory module, so the model can still recover useful signals from it when needed. Across several long-context tasks and different language models, IndexMem reduces memory use while maintaining better accuracy than previous methods, especially when the available memory budget is very limited. This suggests that learned memory management can make long-context language models more practical and reliable.