Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

Autoregressive video diffusion models rely on a KV cache of previously generated frames to avoid redundant computation, but this cache quickly becomes a memory bottleneck as videos grow longer. Methods that quantize the KV cache to low bitwidths reduce memory pressure but degrade video quality. We show that a key driver of this degradation is a systematic bias in the cached partition sum: due to the convexity of the exponential in softmax attention, quantization noise inflates the contribution of cached keys, a phenomenon we call the Jensen bias. This effect causes quantized keys to steal attention mass from the unquantized current tokens. We derive a per-attention-score correction that removes this bias in expectation, computed on the fly from quantization step sizes and query statistics; a second-order Taylor approximation makes the correction cheap and requires no additional KV-cache storage. Evaluated on MAGI-1, SkyReels-V2, and HY-WorldPlay at INT2 quantization, our correction improves PSNR by up to 5.9 dB and raises VBench Score by 7.8 points, recovering much of the quality lost to quantization.