Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

Chunk-wise autoregressive video diffusion models rely on a KV cache of previously generated chunks to avoid redundant computation, but this cache quickly becomes a memory bottleneck as videos grow longer. This bottleneck is a central obstacle for efficient multimodal AI systems, where video and world models must maintain long temporal context for interactive simulation, planning, and embodied agents. Methods to quantize the KV cache to low bitwidths reduce memory pressure but degrade video quality. We show that a key driver of this degradation is a systematic bias in the cached partition sum: due to the convexity of the exponential in softmax attention, quantization noise inflates the contribution of cached keys, a phenomenon we call the Jensen bias. This effect causes quantized keys to steal attention mass from the unquantized current chunk. We derive a per-attention-score correction that removes this bias in expectation, computed on the fly from the quantization step sizes of the cached keys and the query norm. Using a second-order Taylor approximation, the additional computational overhead is negligible, and no additional memory is needed alongside the cache. Evaluated on MAGI-1 and HY-WorldPlay at INT2 quantization, our correction improves PSNR by up to 5.9~dB and raises VBench Score by 7.8 points, recovering much of the quality lost to quantization.