Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core Subspace

Model merging offers an efficient solution for integrating task-specific knowledge from multiple fine-tuned models. Most existing approaches focus on manipulating the difference vectors between fine-tuned and pretrained weights, often overlooking the generalization capabilities inherent in the pretrained parameters. In this work, we revisit the role of pretrained weights in model merging and investigate their efficacy from a subspace perspective. We find that the components of pretrained weights residing in the core subspace—defined by the dominant singular vectors—are essential for maintaining generalization across diverse tasks. Specifically, we present empirical evidence that pretrained weights are nearly first-order stationary and exhibit predominantly non-negative curvature within this core subspace with respect to multi-task loss landscapes, indicating near-optimality. These findings suggest that task-specific adaptations should be injected primarily into the orthogonal complement of the core subspace, thereby preserving the generalization properties of the pretrained model. Extensive experiments on vision and vision-language tasks show that this subspace-aware strategy consistently yields improvements over state-of-the-art training-free merging methods, including Task Arithmetic, LOT Merging, ISO, and TSV. The source code is available at https://github.com/SunWenJu123/model-merging.

How can we combine multiple fine-tuned AI models without damaging the general knowledge learned during pretraining? In this paper, we show that pretrained models already contain highly effective and transferable representations in their most important parameter directions. Changing these directions during model merging can hurt overall performance. Based on this insight, we propose a simple method called Core Subspace Preservation (CSP). Instead of modifying the most important components of the pretrained model, CSP only applies task-specific updates to less critical directions. This helps preserve the model’s generalization ability while still integrating knowledge from multiple tasks. Our experiments on vision, vision-language, and language models show that CSP consistently improves existing model merging methods without requiring extra training data or additional retraining. These findings suggest that preserving the core structure of pretrained models is important for building more reliable and robust multi-task AI systems.