PatternKV: Flattening KV Representation Expands Quantization Headroom

KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness and thus maintains a wide quantization range. Prior work focuses on isolating outliers, which caps their error but fails to flatten the overall distribution, leaving performance fragile under low-bit settings. In this work, we show that the K cache maintains a stable, context-evolving structure, while the V cache carries latent semantic regularities, with both contributing to the organization of vectors into shared patterns. Building on these insights, we propose **PatternKV**, a pattern-aligned residual quantization scheme. It mines representative pattern vectors online, aligns each KV vector to its nearest pattern, and quantizes only the residual. This reshaping of the KV distribution flattens the quantization target and narrows its range, thereby improving the fidelity of low-bit KV quantization. Across long-context and test-time scaling settings on multiple backbones, PatternKV delivers consistent 2-bit gains, incurs only a 0.08\% average 4-bit drop relative to FP16, and improves test-time scaling accuracy by 10\% on average. Compared with FP16, PatternKV further raises throughput by 1.5$\times$ and supports 1.25$\times$ larger batches.

Large language models can answer long questions and solve complex problems by remembering information from earlier words. This memory, called the key-value cache, helps avoid repeated computation, but it can become very large when the input is long or when the model reasons for many steps. A common way to save memory is to store this cache with fewer bits, but doing so can noticeably hurt accuracy because the stored numbers are uneven and difficult to compress. We propose PatternKV, a method that makes this memory easier to compress. Instead of compressing each cache vector directly, PatternKV first finds common patterns among cache vectors, matches each vector to a nearby pattern, and then compresses only the remaining difference. This makes the numbers being compressed smaller and more regular. As a result, PatternKV preserves model quality better under very low-bit compression, especially for long-context and long-reasoning tasks. This can help serve large language models with less memory and higher throughput.