Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success

Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient L2 distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find that the drivers of merge success vary across architectures and merging methods, overall with only a moderate agreement (64.0% average top-5 metric overlap; 79.3% sign agreement). Crucially, however, gradient alignment metrics consistently emerge as the most fundamental signals of mergeability. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.

Model merging is a technique to fuse multiple models into one that is able to solve a multitude of tasks. However, when merging models, what determines their compatibility (mergeability) is underexplored. We find that mergeability is not an intrinsic property of a model but a function of model expertise, merging method, and partner models to merge with. We discover this via an interpretable and flexible linear framework that, for each merging method, unveils specific properties that best correlate to mergeability, showing that these properties vary largely across different merging methods. These results pave the way for merge-aware finetuning strategies that specifically encourage these properties during training time.