ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection

Long-context inference is constrained by KV-cache memory, which grows linearly with sequence length; KV-cache compression therefore hinges on reliably selecting which past tokens to retain. Most geometry-based eviction methods score keys by cosine similarity to a global centroid, but cosine is scale-invariant and can discard magnitude cues that distinguish semantically salient tokens. We propose ManifoldKV, a training-free scorer that ranks tokens by Euclidean distance to the key centroid, capturing both angular and radial deviations. On the RULER benchmark, ManifoldKV achieves 95.7% accuracy at 4K–16K contexts with 20% compression, matching the best geometric baseline overall while decisively outperforming it in two regimes where magnitude information is critical. First, on multi-key retrieval, ManifoldKV reduces directional collisions, achieving 92.4% vs KeyDiff's 77.0% (+15.4 points) on 3-key NIAH at 50% compression. Second, to address dilution and performance collapse of global centroids at 64K context, we introduce WindowedManifoldKV, which restores accuracy to 84.3% at 25% compression, a 49-point recovery over global L2 and +3.2 points over KeyDiff. Beyond RULER, we validate on real-world benchmarks: on LongBench, ManifoldKV outperforms KeyDiff by +2.80 points on Qwen3-8B (winning 12 of 14 tasks) and +0.49 on Phi-4; on HELMET, ManifoldKV achieves +6.5 EM on RAG and WindowedManifoldKV reaches +42 points on multi-key recall at 131K; and on InfiniteBench at 100K+ context, WindowedManifoldKV wins by +7.16 on Phi-4. Cross-architecture evaluation across six models reveals that the optimal distance metric depends on key-norm geometry, providing the first systematic guidelines for metric selection in geometric KV cache compression. The method requires only 3 lines of code and works across diverse architectures without tuning.

Modern AI assistants must "remember" everything they have read in a long document or conversation. The longer the text, the more memory this takes — enough to become a serious speed and cost bottleneck. To cope, systems discard part of what they have memorized, but dropping the wrong pieces makes the AI forget important details like names, numbers, and dates. We found that the genuinely important pieces stand out in the model's internal "map" of the text: they sit unusually far from the ordinary, run-of-the-mill pieces. Earlier methods only checked whether a piece pointed in an unusual direction, and overlooked pieces that were unusual in magnitude. Our method, ManifoldKV, measures the full distance instead, so it reliably keeps the standout pieces that matter — in about three lines of code, with no retraining, and across many different models. As a result, AI systems can handle very long inputs with far less memory while still recalling the details that count, especially when several facts must be retrieved at once or the text is extremely long. We also give practitioners a simple rule for which version of the measurement to use on a given model.