Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs

Quantization Error Reconstruction (QER) reduces accuracy loss in Post-Training Quantization (PTQ) by approximating weights as $\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$, using a rank-$r$ correction to reconstruct quantization error. Prior methods devote the full rank budget to error reconstruction, which is suboptimal when $\mathbf{W}$ has intrinsic low-rank structure and quantization corrupts dominant directions. We propose Structured Residual Reconstruction (SRR), a rank-allocation framework that preserves the top-$k$ singular subspace of the activation-scaled weight before quantization, quantizes only the residual, and uses the remaining rank $r-k$ for error reconstruction. We derive a theory-guided criterion for selecting $k$ by balancing quantization-exposed energy and unrecoverable error under rank constraints. We further show that resulting $\mathbf{Q}+\mathbf{L}\mathbf{R}$ parameterization naturally supports Quantized Parameter-Efficient Fine-Tuning (QPEFT), and stabilizes fine-tuning via gradient scaling along preserved directions. Experiments demonstrate consistent perplexity reductions across diverse models and quantization settings in PTQ, along with a 5.9 percentage-point average gain on GLUE under 2-bit QPEFT. The project page is available at https://ai-isl.github.io/srr.

Large language models (LLMs) require enormous memory and computation, making deployment expensive. A common solution is quantization, which converts precise numerical weights into compact low-bit representations. However, aggressive quantization can significantly degrade model accuracy. To reduce this damage, prior methods often add low-rank matrices — compact correction modules that use only a small number of additional parameters — to recover the errors introduced by quantization. The problem is that this correction capacity is limited, and existing approaches spend all of it fixing errors after quantization, even though some important information has already been lost. We propose a new preserve-then-quantize approach that rethinks how this limited correction capacity is used. Instead of quantizing the entire weight matrix and repairing errors afterward, our method first identifies the most important directions and preserves them in higher precision. Only the remaining, less critical part is quantized, while the remaining correction capacity is used to reconstruct the resulting errors. A lightweight theoretical criterion automatically determines, for each layer, how much capacity should be used for preservation versus reconstruction. It also naturally extends to fine-tuning: because the low-rank correction is already part of the model, it can be directly optimized for downstream tasks.