Mitigating Error Propagation in Low-Rank Approximation of Large Models via Distribution-Aware Whitening

Low-rank approximation has emerged as a cornerstone technique for model compression and parameter-efficient fine-tuning, enabling substantial reductions in computation and memory without altering model architectures. However, existing approaches often overlook the shifts in feature distributions induced by the approximation process, which can lead to error amplification and unstable inference. We propose a distribution-aware whitening framework that dynamically whitens layer inputs based on the evolving feature distributions, ensuring second-order isotropy of input features. This allows that the discarded components in the low-rank approximation are those with minimal impact on model outputs, thereby minimizing cumulative approximation errors across layers. We theoretically analyze how distribution misalignment leads to error propagation and demonstrate that our approach achieves tighter control over layerwise distortion. Extensive experiments across various large language models demonstrate the superiority of our method in post-training compression. Moreover, our method can also serve as an effective initialization for LoRA-style parameter-efficient fine-tuning. Our findings highlight the importance of considering feature distributions in low-rank approximations, paving the way for reliable and effective model compression strategies.

Large AI models are powerful but often require enormous memory and computing resources, making them difficult to deploy efficiently. A common solution is to simplify these models using “low-rank approximation,” a technique that reduces the number of parameters while trying to preserve performance. However, existing methods often ignore how this simplification changes the internal data representations inside the model, which can cause errors to accumulate and reduce reliability. In this work, we introduce a new method that adapts to these changing feature distributions during compression. Our approach continuously normalizes internal representations so that the compression process removes only the least important information while preserving the model’s behavior as much as possible. We also provide theoretical analysis explaining why mismatched feature distributions can lead to error amplification. Experiments on several large language models show that our method achieves better compression performance than existing approaches. In addition, it can improve the initialization of lightweight fine-tuning methods such as LoRA. Our results suggest that understanding and controlling feature distributions is important for building more efficient and reliable AI systems.