BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks—often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries—compact representatives for each global query cluster—to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to $5.8\times$ memory reduction while nearly preserving full cache accuracy and improving throughput by over $4.3\times$.

Large reasoning models can solve hard math, science, and coding problems by writing long step-by-step answers before giving a final response. But as they write, they must also keep a growing memory of earlier words, and this memory can become so large that it limits how many users or tasks can fit on specialized computer chips. We found that during long reasoning, these models sometimes look far back to earlier parts of their own answer, such as the original plan or problem constraints, instead of only looking at the most recent words. Existing memory-saving methods often miss this behavior and may delete information that the model needs later. Our method, BeaconKV, keeps a small set of beacon signals that act like signposts for important earlier reasoning paths. These beacons help the model decide which earlier information to keep, without retraining the model or storing its full history. Across several open-source reasoning models and benchmarks, BeaconKV reduces memory use by up to $5.8\times$ while keeping accuracy close to full-memory inference and improving processing speed by over $4.3\times$. This can make advanced reasoning models cheaper and easier to deploy on limited hardware.