Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project Page & Code: https://lems-svd.github.io

Large language models (LLMs) powering modern chat apps are incredibly good at answering questions, but they rely on billions of parameters spread across hundreds of layers. This massive size makes them expensive and difficult to run. While compression techniques can shrink these models, figuring out exactly how much to compress each specific layer is a complex combinatorial problem. To solve this, we developed an algorithm called Layer-wise Error Modeling Search (LEMS). LEMS models how errors accumulate throughout the network to effectively distribute the compression budget (e.g. 60 % of the original size). We paired this with a revamped layer-wise compression technique that analyzes text word-by-word rather than sentence-by-sentence. Surprisingly, this simple shift bypasses a common mathematical bottleneck that prevents a similar model from achieving high performance. Together, these techniques successfully compress LLMs into a much smaller footprint while preserving their impressive answering capabilities better than previous methods.