Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose Zero Sum SVD (ZS-SVD), a post-training method that performs global singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. ZS-SVD prunes components across the whole model with a zero sum rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a single projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD

Large language models are powerful, but they are expensive to store and run. This makes it hard to deploy them on devices or systems with limited memory, such as edge devices, robotics platforms, or low-cost servers. One way to reduce this cost is to replace large model matrices with smaller low-rank versions, but existing methods often compress similar matrices in the same way, even when some parts of the model are much more important than others. We propose Zero-Sum SVD, a post-training compression method that measures how important different matrices and their components are to the model’s performance. Instead of applying the same compression level everywhere, our method estimates how much removing each component would affect the model’s loss. It then compresses the model based on these importance estimates, removing less important components more aggressively while preserving the parts that matter more. We also add a lightweight correction step that helps recover performance after compression. Across several large language models, Zero-Sum SVD preserves accuracy and perplexity better than prior SVD-based compression methods while also improving inference speed. This makes large models cheaper and more practical to use without training a new model from scratch.