Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective

Key–value (KV) caching is essential for large language model inference, yet its memory overhead poses a critical bottleneck for long-context generation. Existing eviction policies predominantly rely on empirical heuristics, lacking a rigorous theoretical foundation. This work rethinks KV cache eviction through the lens of the Information Bottleneck principle. Under a linear–Gaussian surrogate of attention, we derive a closed-form mutual information objective that characterizes the effective information capacity of a retained KV cache subset. This formulation reveals that a wide range of existing eviction strategies can be interpreted as different approximations of the same capacity-maximization principle. Guided by this insight, we introduce CapKV, a capacity-aware eviction method that directly targets information preservation via a log-determinant approximation using statistical leverage scores. This approach replaces heuristic selection with a theoretically grounded mechanism that preserves the maximum predictive signal. Extensive experiments across multiple models and long-context benchmarks show that CapKV consistently outperforms prior methods, achieving a better trade-off between memory efficiency and generational fidelity.

Large Language Models (LLMs) require an immense amount of computer memory to process long text documents, creating a critical bottleneck that slows down generation and drives up infrastructure costs. To save memory, current systems routinely throw away intermediate data—a process called "cache eviction"—but they do so using rule-of-thumb heuristics that lack a solid theoretical foundation. In this work, we fundamentally rethink this problem by viewing memory management through the lens of information communication theory. We treat the stored memory as a communication channel and mathematically prove that existing methods are simply different approximations of a single, grand goal: maximizing the relevant information passed to future steps. Driven by this insight, we introduce CAPKV, a smart memory-retention method that calculates the unique informational value of each data point and safely discards redundant ones. Extensive experiments prove that CAPKV drastically reduces memory usage while preserving the AI's generation quality, providing a more stable and theoretically grounded path toward efficient, long-context AI applications.