Representation Drift Compensation: A Near-Zero Inference Cost Enhancement for LLM Decomposition

While low-rank decomposition offers potential for reducing LLM parameters, maintaining the original capabilities remains a significant challenge. In this work, we identify and formalize a key overlooked issue in LLM decomposition: \textit{representation drift}. We show that approximation errors introduced by decomposition propagate and amplify non-linearly through the deep layers of the transformer architecture, progressively distorting internal representations and degrading downstream performance. To mitigate this, we introduce a conceptually simple but principled compensation mechanism, named ``Decomper'', that operates by suppressing error at its source. By learning to align the output distribution of decomposed transformer blocks with their original counterparts, our method effectively counteracts representation drift, achieving notable performance recovery with near-zero inference overhead. Extensive experiments in OPT, LLaMA-2/3, and Qwen exhibit remarkable improvements. For instance, on LLaMA-3-8B and OPT-13B at 40\% compression, perplexity is reduced by more than 70\% while reasoning task accuracy improves by over 10\%. Our code is available at this \href{https://github.com/xinhaoH/SVD-Compen}{URL}.

Large Language Models (LLMs) are incredibly capable but require massive computing power and memory. To make them cheaper and faster to run, researchers often "compress" them by shrinking their massive mathematical matrices into smaller approximations (a technique called low-rank decomposition). However, this compression often degrades the model's intelligence and accuracy. In this paper, we uncover why this performance drop happens: a phenomenon we call "representation drift." Because LLMs are deep networks, a tiny approximation error in an early layer acts like a bad translation in a game of telephone. As the data passes through dozens of layers, these tiny errors snowball and multiply, eventually distorting the model's internal signals so badly that the final output is ruined. To solve this, we introduce a simple but effective ''compensation'' mechanism. Instead of just letting the errors accumulate, we learn a tiny correction term (a bias) for each compressed layer. This acts like a localized error-checker, suppressing the mistake at its source before it has a chance to spread and amplify. Because this correction is fused into the model's existing structure, it restores performance with zero extra cost to the model's running speed.