Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

While Key-Value (KV) cache compression is essential for efficient LLM inference, current evaluations disproportionately focus on \textbf{sparse retrieval} tasks, potentially masking the degradation of High-Density Reasoning where Chain-of-Thought (CoT) coherence is critical. We introduce KVFundaBench to systematically evaluate this gap, revealing a sharp dichotomy: while retrieval tasks remain robust, reasoning tasks exhibit severe Task-Dependent Degradation under aggressive compression due to disrupted CoT links. Extending our analysis to the DeepSeek-R1 model, we uncover that its specialized attention patterns offer unique insights into the fragility of reasoning chains. Guided by these findings—specifically the necessity of preserving few-shot examples as indivisible \textbf{Semantic Units}—we propose ShotKV. This approach explicitly separates prefill and decoding phases to prioritize semantic integrity. Empirical results demonstrate that ShotKV achieves 9\%-18\% accuracy improvements on long-context generation tasks and effectively generalizes to document QA, all while delivering an 11\% latency reduction compared to full cache inference.

Modern AI assistants need a kind of short-term memory to follow long conversations and read lengthy documents. This memory grows quickly and uses a lot of computing power, so researchers shrink it to make AI cheaper and faster to run. Until now, this shrinking has mostly been tested on tasks like finding a specific fact buried in a long passage — essentially looking for a needle in a haystack. We show this is misleading. When an AI is asked to solve a problem step by step, such as working through math, even modest shrinking breaks the chain of reasoning and causes the AI to make many more mistakes. We built a test suite that systematically measures this damage across math, knowledge, common-sense, and safety questions, and we propose a simple fix: instead of cutting individual words to save memory, we keep each worked example whole. This change recovers most of the lost accuracy and even speeds up the AI by about 11%, making capable AI both more reliable and more affordable to deploy.