OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generation tasks. However, their massive parameter scale leads to significant resource consumption and latency during inference. Post-training weight-only quantization offers a promising solution by reducing model size and accelerating token generation through alleviating the memory-bound issue. Nevertheless, there are inherent systematic outliers in weights, and although some efforts have attempted to address them, such as scaling and rotation, the performance of low-bit quantization remains far from satisfactory. In this paper, we propose Outlier Self-Absorption Quantization (OSAQ), which performs second-order low-rank derived additive weight suppression for low-bit weight-only LLM quantization. Specifically, we observe that Hessian exhibits low-rank consistency across different inputs, with certain directions persistently lacking strength. Leveraging this property, we construct an additive weight transformation based on the Hessian’s null space, thereby suppressing weight outliers without affecting the task loss. This additive transformation can be absorbed into the weights offline, requiring no inter-layer transformations and introducing no inference overhead. Moreover, the construction is efficiently achieved by a closed-form solution, without resource-intensive training or iterative procedures. Extensive experiments across models of varying scales and tasks are conducted, and the results show that OSAQ effectively suppresses outliers and improves low-bit quantization performance.

This paper presents a new way to make large language models (LLMs) smaller, faster, and more efficient without significantly reducing their performance. Modern LLMs require enormous computing resources because they contain billions of parameters, making them expensive to run and difficult to deploy on everyday devices. One common solution is to compress the model by reducing the precision of its stored numbers, but this often harms accuracy because some unusually large parameter values are difficult to compress effectively. We propose a method called Outlier Self-Absorption Quantization (OSAQ) to address this challenge. Instead of relying on costly retraining or complicated processing steps, OSAQ identifies parts of the model that can be safely adjusted without affecting its behavior. By carefully reducing problematic extreme values in the model weights, our method improves the effectiveness of low-bit compression while preserving performance. Importantly, these adjustments are applied only once before deployment, so the method adds no extra cost during actual use. Experiments on language models of different sizes and across multiple tasks show that OSAQ consistently improves low-bit model compression and enables more efficient inference. This work helps make advanced AI models more practical for real-world applications, especially in environments with limited computing resources such as personal devices and edge systems.