1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization

Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, the optimal choice of quantization format and bit-width presents a challenge in practice. The full design space of quantization is not fully explored in the context of QAT, and the precise trade-off between quantization and downstream performance is poorly understood, as comparisons often rely solely on perplexity-based evaluations. In this work, we address these shortcomings with an empirical study of QAT in the low-bit regime. We show that k-means based weight quantization outperforms integer formats and can be implemented efficiently on standard hardware. Furthermore, we find that, under a fixed inference memory budget, the best performance on generative downstream tasks is achieved with $1$-bit quantized weights.

Large language models can answer questions, write code, and follow instructions, but running them is expensive because their billions of internal numbers must be stored and moved through computer memory. A common way to reduce this cost is to store each number in lower precision, usually by rounding it to one of a small set of allowed values; for example, 0.04324 might become 0.04. The hard question is how far we can shrink these numbers before the model becomes much worse, and whether it is better to keep a smaller model in high precision or fit a much larger model into the same memory by using very low precision. We study this question by training and comparing language models under matched memory budgets. Instead of forcing weights into evenly spaced values, we use a simple clustering method that learns better value choices for the model. Across our experiments, this approach works better than standard low-bit formats. Most importantly, when the memory budget is fixed, a much larger 1-bit model, where each weight is reduced to one of two values, roughly like −1 or +1, performs best on many real generation tasks, including reasoning and coding. These results suggest that future efficient language models may get more value from using memory to store more parameters, rather than storing each parameter more precisely.