When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

Model merging combines multiple fine-tuned models into a single model by $\textit{adding}$ their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across vision and language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0\%. Code is available at https://github.com/lyymuwu/SVC.

Modern AI systems are often adapted to different tasks separately, such as recognizing different types of images or solving different language problems. A practical way to build a more capable system is to combine these task-specific models into one model, without retraining everything from scratch. However, we find that this combination process can sometimes count the same shared knowledge too many times. As a result, the combined model may become overly focused on a few common patterns and fail to preserve some task-specific abilities. This paper proposes a simple way to detect and correct this problem after models have already been combined. Our method adjusts how strongly the combined model relies on different internal directions, without requiring any training data or additional optimization. This makes model merging more reliable and efficient. Experiments on vision and language tasks show that our method improves several existing model merging approaches, helping the final model better balance shared knowledge and task-specific skills.