Fast KV Compaction via Attention Matching

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through *compaction* in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges (Eyuboglu et al., 2025) has shown that it is possible to *train* highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for *fast* context compaction in latent space through **Attention Matching**, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to $50\times$ compaction in seconds on some datasets with little quality loss.

When an AI language model reads a long document or holds a long conversation, it keeps an internal memory of everything it has seen. This memory grows with the length of the input, and for long texts it becomes a major bottleneck — taking up large amounts of computer memory and slowing the system down. A common fix is to summarize the text into something shorter, but summaries discard details that may later turn out to matter, which hurts the model's answers. A recent alternative compresses this memory directly and preserves quality well, but it is slow and expensive, often needing hours of computation for a single document. We introduce a method called Attention Matching that compresses the internal memory quickly while keeping the information the model actually relies on. Rather than retraining anything, it builds a compact memory designed to reproduce how the model would have behaved on the original text, using simple calculations that finish in seconds. This makes it possible to shrink the memory many times over with little loss in accuracy, helping AI systems handle long documents and conversations far more efficiently and affordably.