CALM: Consensus-Aware Localized Merging for Multi-Task Learning

Model merging aims to integrate the strengths of multiple fine-tuned models into a unified model while preserving task-specific capabilities. Existing methods, represented by task arithmetic, are typically classified into global- and local-aware methods. However, global-aware methods inevitably cause parameter interference, while local-aware methods struggle to maintain the effectiveness of task-specific details in the merged model. To address these limitations, we propose a Consensus Aware Localized Merging (CALM) method which incorporates localized information aligned with global task consensus, ensuring its effectiveness post-merging. CALM consists of three key components: (1) class-balanced entropy minimizationsampling, providing a more flexible and reliable way to leverage unsupervised data; (2) an efficient-aware framework, selecting a small set of tasks for sequential merging with high scalability; (3) a consensus-aware mask optimization, aligning localized binary masks with global task consensus and merging them conflict-free. Experiments demonstrate the superiority and robustness of our CALM, significantly outperforming existing methods and achieving performance close to traditional MTL.

This work investigates effective methods for merging multiple fine-tuned task-specific models into a unified multi-task model. Existing approaches either optimize merging parameters for individual tasks, inevitably introducing parameter interference, or focus solely on extracting localized parameters for each task, which may lack generalizability across tasks. To address these limitations, we propose a Consensus-Aware Localized Merging (CALM) framework. Our key insight is that localized parameter selection should align with global task-level consensus to ensure compatibility and effectiveness in multi-task scenarios. CALM consists of three key components: (1) class-balanced entropy-minimization sampling, which provides a flexible and reliable mechanism to leverage unlabeled data; (2) an efficiency-aware sequential merging framework that selects a minimal subset of tasks for scalable integration; (3) consensus-aware mask optimization, which aligns localized binary masks with global task consensus and enables conflict-free merging. Experiments demonstrate that CALM significantly outperforms existing model merging methods and achieves performance comparable to traditional multi-task learning, while maintaining superior scalability and robustness.