UniSVQ: 2-bit Unified Scalar-Vector Quantization

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.

AI language models like ChatGPT are powerful but expensive to run, since a mid-size model can require more GPU memory than a typical workstation has available. Shrinking models to use fewer bits per value dramatically cuts memory and speeds up responses, but current 2-bit compression methods force an uncomfortable choice: fast methods (scalar quantization) lose too much quality, while accurate methods (vector quantization) are too slow for real-world use. We developed UniSVQ, a 2-bit compression method that gets the best of both worlds. The key idea is a compact mathematical shortcut that lets the model stay accurate like the high-quality methods while running as fast as the simpler ones. We also include an automatic tuning step that recovers the quality loss after compression. Across a wide range of AI models, UniSVQ matches the accuracy of the best existing methods while running 1.68× faster than the uncompressed model and using one-eighth the memory. This makes it practical to deploy powerful AI models on everyday hardware, lowering the cost and energy footprint of AI for both researchers and end users.