FairMerging: Rethinking Model Merging through the Lens of Fairness

Model merging offers an appealing route to multi-task learning by composing independently fine-tuned checkpoints without centralized data or retraining. However, this convenience can come with a hidden cost. Model merging may amplify performance disparities across subgroups, raising fairness concerns even when average accuracy remains competitive. To explain this phenomenon, we develop a sensitivity-based theoretical analysis that upper bounds the fairness gap induced by model merging. Theoretical analysis with empirical verifications reveals that the resulting fairness gap is governed by two coupled factors, a merging magnitude term that measures how far the merged parameters move from the target model and global sensitivity terms that determine how unevenly the perturbation affects subgroup losses. Guided by these insights, we propose FairMerging, a two-stage merging framework that first reduces the sensitivity of the target model and then performs fairness-aware coefficient optimization with orthogonally normalized task vectors. Experiments across multiple datasets, backbones, and merging baselines demonstrate that FairMerging substantially mitigates unfairness while retaining competitive multi-task performance.

Modern machine learning models are often adapted to many different tasks, but training one large model jointly on all tasks can require shared data, high compute, and repeated retraining. Model merging offers a cheaper alternative: it combines several separately fine-tuned models into one model by mixing their parameter updates. This paper shows that this convenience can create a hidden fairness problem. Even when the merged model performs well on average, it may become much worse for some subgroups than for others. We study why this happens and find that unfairness after merging depends on two main factors: how far the merged model moves away from the target model, and how differently each subgroup reacts to that change. Guided by this analysis, we propose FairMerging, a two-stage method for fairness-aware model merging. It first adjusts the target model so that different subgroups respond more evenly to parameter changes. It then chooses merging coefficients while explicitly considering both task performance and fairness risk. Experiments on multiple vision datasets and model backbones show that FairMerging reduces subgroup performance gaps while keeping competitive multi-task accuracy. This suggests that model merging should be evaluated not only by average accuracy, but also by how it affects different groups.