SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights

Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision. Still, current methods show perplexity degradation at bit-widths $\leq 4$, partly because representing outliers causes precision issues in parameters that share the same scales as these outliers. This problem is especially pronounced for calibration-free, uniform quantization methods. We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn–Knopp–style algorithm that finds scales to normalize per-row and per-column variances. We show that this approximates activation-aware quantization by recovering column scales from the weight matrix structure that are predictive of the typical activation magnitudes the matrix received during training. Our method has no interactions between layers and can be trivially applied to new architectures to quantize any linear layer. We evaluate our method on the Qwen3 model family, among others. SINQ reduces the perplexity gap on WikiText2 and C4 by over 50% against uncalibrated uniform quantization baselines, incurs zero to negligible compute overhead, and can be further enhanced by combining it with calibration and non-uniform quantization levels. Code is available in the supplementary.

Chatbots based on LLMs typically store their 'knowledge' in 16-bit parameters. We propose a method to reduce this precision to 4 bits while keeping the 'knowledge' intact. This approximately cuts the memory required to run a given LLM down by a factor of four. Previous methods to approach this, required sample data (a 'calibration set') and a lot of computation. For our method, all one needs to know is inferred from the parameters. This is faster and avoids the problem that the LLM may over-specialize on this calibration set. We also show that our method can be combined with such 'calibrated' approaches and other existing methods.