xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction

Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache across layers, but these approaches either require expensive pretraining or rely on per-token cross-layer cosine similarity that is often limited in practice. We show, via Centered Kernel Alignment (CKA), that the dominant singular vectors of KV-Cache are well aligned across layers. Motivated by this observation, we propose xKV, a post-training compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, substantially reducing KV-Cache memory. Across widely used LLMs, xKV achieves up to 8× KV-Cache compression while preserving accuracy on long-context tasks and in multi-turn settings. To further improve efficiency, we introduce Selective Reconstruction (SR) at decode time. Combined with SR, xKV achieves up to 4.23× end-to-end speedup over the full attention baseline, and surpasses notable baselines with 30% higher throughput under a similar accuracy level. Overall, xKV provides a plug-and-play approach to reduce both memory and latency for long-context LLM inference. Our code is publicly available at: https://github.com/abdelfattah-lab/xKV.

LLMs can now read enormous inputs at once, like entire books or codebases. To work efficiently, a model keeps a running memory of everything it has seen, but for long inputs, this memory can become huge, limiting how many users a system can serve and slowing it down. Prior fixes either retrain the model from scratch or compress each layer alone. We found that different layers store this memory using the same underlying representation, even though it looks different on the surface. Our method, xKV, keeps one shared representation across layers, cutting memory up to eight times with minimal accuracy loss and several times faster responses, all without retraining.