Pushing the Limits of Block Rotations in Post-Training Quantization

Recent post-training quantization (PTQ) methods have adopted block rotations to diffuse outliers prior to rounding. While this reduces the overhead of online full-vector rotations, the effect of block structure on outlier suppression remains poorly understood. To fill this gap, we present the first systematic, non-asymptotic analysis of outlier suppression for block Hadamard rotations. Our analysis reveals that outlier suppression is fundamentally limited by the geometry of the input vector. In particular, in the deterministic worst case, post-rotation outliers are minimized when the pre-rotation $\ell_1$ norm mass is evenly distributed across blocks. Guided by these insights, we introduce PeRQ (Permute, Rotate, then Quantize), a PTQ framework that redistributes activation mass via permutations prior to rotation. We propose a greedy mass diffusion algorithm to calibrate permutations by equalizing the expected blockwise $\ell_1$ norms. To avoid adding inference overhead, we identify permutation-equivariant regions in transformer architectures to merge these permutations into model weights before deployment. Experiments show that PeRQ consistently improves accuracy across all block sizes, recovering up to 90% of the full-vector rotation perplexity when quantizing Llama3 1B to INT4 with block size 16, compared to 46% without permutations.

Large language models are powerful but expensive to deploy, so engineers compress their weights and activations into low-precision datatypes, a process called quantization. A handful of unusually large activation values, called outliers, dominate the numerical range and degrade accuracy at low bit widths. A popular remedy is to redistribute activation magnitudes across coordinates by rotating the tensors before rounding. Rotating the full activation vector works best but is computationally expensive at inference time, so recent methods partition activations into fixed-size blocks and rotate each block independently. Until now, it has been unclear why this block structure sometimes fails to suppress outliers. We provide the first deterministic analysis of outlier suppression under block rotations, showing that suppression is fundamentally limited by how activation magnitude is distributed across blocks before rotation, and is most effective when that the pre-rotated activation mass is evenly balanced across blocks. Guided by this insight, we introduce PeRQ (Permute, Rotation, then Quantize), which calculates a permutation that balances activation magnitudes before rotation. By exploiting equivariance properties within transformer architectures, this permutation can be absorbed into surrounding weights, recovering accuracy at no additional inference cost.