NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs

Large language models (LLMs) achieve impressive performance across domains but face significant challenges when deployed on consumer-grade GPUs or personal devices such as laptops, due to high memory consumption and inference costs. Post-training quantization (PTQ) of LLMs offers a promising solution that reduces their memory footprint and decoding latency. In practice, PTQ with uniform quantization representation is favored due to its efficiency and ease of deployment, as uniform quantization is widely supported by mainstream hardware and software libraries. Recent studies on low-bit uniform quantization have led to noticeable improvements in post-quantization model performance; however, they mainly focus on quantization methodologies, while the initialization of quantization parameters remains underexplored and still relies on the conventional Min-Max formula. In this work, we identify the limitations of the Min-Max formula, move beyond its constraints, and propose NeUQI, a method that efficiently determines near-optimal initialization for uniform quantization. Our NeUQI simplifies the joint optimization of the scale and zero-point by deriving the zero-point for a given scale, thereby reducing the problem to a scale-only optimization. Benefiting from the improved quantization parameters, our NeUQI consistently outperforms existing methods in the experiments with the LLaMA and Qwen families on various settings and tasks. Furthermore, when combined with a lightweight distillation strategy, NeUQI even achieves superior performance to PV-tuning, a considerably more resource-intensive method.

Large language models are useful for tasks like writing, coding, and answering questions, but they are hard to run on laptops and consumer GPUs because they use large amounts of memory and computation. A common way to make them smaller and faster is to represent their numbers in low-bit integer (INT) formats. These INT formats are practical because they are widely supported by today’s hardware and software. However, many current methods still choose the starting settings for this conversion using a simple rule based only on the smallest and largest values in the model, which can cause unnecessary accuracy loss. We propose NeUQI, a method for choosing better starting settings when converting large language models to INT formats. NeUQI simplifies this choice by reducing a two-part setting problem to a simpler search over one main setting. In experiments with LLaMA and Qwen models, NeUQI improves accuracy across different tasks and compression settings. When combined with a small amount of distillation, it can even outperform a much more resource-intensive tuning method.