Principled SVD-based Delta Compression via Quantization Error Minimization

Supervised Fine-Tuning (SFT) empowers Large Language Models (LLMs) with exceptional performance on specialized tasks, but it yields dense, high-dimensional delta parameters that pose severe storage and distribution challenges. Singular Value Decomposition (SVD)-based compression offers a compact representation for such delta parameters, but existing methods adopt heuristic quantization without clarifying underlying mechanisms, leading to poor generalizability. In this work, we propose PrinMix, a rigorous SVD-based framework that models quantization as an optimization problem, grounding the design in mathematical mechanisms. We first theoretically derive quantization error and identify a key singular-value-dominated scaling mechanism, which mathematically proves the necessity of mix-precision quantization. We then model the quantization scheme as a 0/1 Integer Linear Programming (ILP) problem, which yields optimal bit-budget-constrained solutions without empirical assumptions. Furthermore, PrinMix integrates a Reconstruction Target Correction (RTC) method to compensate for errors from the $\mathbf{V}$-then-$\mathbf{U}$ sequential quantization process. Extensive experiments confirm PrinMix performs well: for 7B LLMs, PrinMix outperforms SOTA Delta-CoMe on challenging benchmarks by 22.3\% on AIME2024 and 6.1\% on GQA.

Many companies and researchers start with a large base model and then adapt it for different jobs, such as solving math problems, writing code, or understanding images. But storing every customized version separately takes a lot of memory and money, which makes it hard to deploy many of them at once. Our work introduces PrinMix, a way to compress only the task‑specific model updates while keeping a single shared base model. Rather than relying on fixed rules or guesswork, PrinMix estimates which parts of those updates matter most and keeps more detail by applying high‑precision bits, while compressing less important parts more aggressively. It also adds a correction step to reduce the information lost during compression. This makes it possible for one shared base model to support many specialized versions at much lower storage cost. In our experiments, PrinMix preserved quality better than earlier update-compression methods, especially on harder tasks. This could make advanced AI systems cheaper to store, easier to serve, and more practical for real-world use.