SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation

Model merging combines independently trained models into a single multi-task model. However, most existing approaches focus primarily on avoiding task interference. We argue that its greater potential lies in enabling task synergy, where tasks actively improve one another. We identify cross-task performance, defined by compatibility between encoders and predictors across tasks, as a key indicator of merge quality. We demonstrate that adapting only a single task-specific layer is sufficient to induce such synergy. This study proposes SyMerge, a lightweight framework that jointly optimizes merging coefficients and a single task-specific layer. We adopt an expert-guided self-labeling objective, providing stable supervision beyond entropy minimization. Intriguingly, we further show that SyMerge successfully merges models trained from different initializations, a regime where standard methods break down. Our minimalist yet principled method achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks. Our code is available at https://aim-skku.github.io/SyMerge.

Modern AI systems are often trained separately for different tasks, but keeping many separate models is expensive. One solution is to combine these trained models into a single model that can handle all tasks at once. However, when models are merged, they often interfere with one another, causing performance to drop. Most existing methods try to reduce this conflict, but we ask a more ambitious question: can merged tasks actively help one another instead? We introduce SyMerge, a lightweight method that encourages cooperation between tasks during model merging. Instead of retraining the whole system, SyMerge adjusts how strongly each model contributes to the merged model and adapts only one small task-specific part. This small adjustment makes the merged model more compatible across tasks, so knowledge from one task can better support another. As a result, SyMerge outperforms previous merging methods across image recognition, language understanding, and visual scene-understanding tasks, and even succeeds when combining models trained from different setups, where previous methods break down.