ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose \textbf{ProjQ}, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to $2\times$ lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on \url{https://github.com/yy9301/ProjQ}.

Large language models are powerful, but they are often too large to run efficiently on limited hardware such as personal devices or edge servers. A common solution is to compress the model by storing its parameters with fewer bits, but this compression introduces errors that can hurt the model’s performance. Another common technique, called low-rank adaptation, adds a small trainable module to help the compressed model recover and adapt to new tasks. However, standard compression methods often create errors that are spread across many directions, making them difficult for this small module to correct. We propose ProjQ, a compression method that deliberately shapes the compression error so that it lies mostly in directions that the small adaptation module can easily fix. Instead of only trying to make the total compression error small, ProjQ also controls the structure of that error. This allows the adaptation module to spend less capacity repairing compression damage and more capacity learning the target task. Experiments on LLaMA2, Qwen2.5, and Qwen3 show that ProjQ improves both error compensation and downstream fine-tuning under very low-bit compression. These results suggest that future model compression methods should consider not only how much error they introduce, but also whether the error is easy for later adaptation to correct.