TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization

4-bit quantization reduces the memory footprint and latency of large language model inference, but its aggressive precision reduction can severely degrade accuracy. Prior methods address this by decomposing each weight matrix into two components (e.g., via singular value decomposition) and quantizing them separately, assigning the bulk of values to a low-precision residual component while handling outliers with a high-precision low-rank component. However, such decompositions are designed to minimize the real-valued energy of the residual, rather than the post-quantization error of the residual and low-rank components. We propose TwinQuant, a 4-bit quantization framework that learns quantization-friendly decomposed subspaces and jointly reshapes both the low-rank and residual components. TwinQuant learns component-specific transformations via a joint optimization over the Stiefel and general linear manifolds, flattening their distributions and reducing dynamic-range imbalance. To enable efficient end-to-end execution, we further design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components with a single epilogue, avoiding intermediate global-memory traffic. Across LLaMA3 and Qwen3 models, TwinQuant preserves near-FP16 accuracy and delivers up to $1.8\times$ end-to-end speedup over an FP16 baseline.

Large language models power tools such as chatbots, coding assistants, and search systems. However, running these models is expensive because they store and process huge amounts of numerical data. A common way to reduce this cost is to store each number with fewer bits, such as 4 bits instead of 16. This can make models faster and smaller, but it often hurts model quality. We introduce TwinQuant, a method that makes this compression more accurate and efficient. Instead of compressing the model numbers directly, TwinQuant separates them into two parts and learns how to arrange both parts so that they can be represented more safely with only 4 bits. It also includes a specialized GPU program that computes these two parts together, so the computer does not waste time moving temporary data around. On LLaMA3 and Qwen3 models, TwinQuant keeps performance close to the original high-precision models while making model execution up to 1.8$\times$ faster.