When Model Merging Breaks Routing: Training-Free Calibration for MoE

Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining. However, existing merging techniques, largely based on linear parameter arithmetic or optimization, struggle when applied to Mixture-of-Experts (MoE) architectures. We identify a critical failure mode in MoE merging, termed *routing breakdown*, in which the merged router fails to dispatch tokens to suitable experts. Routing breakdown stems from the sensitivity of the non-linear softmax and discrete Top-$k$ routing mechanisms to parameter perturbations from merging, a sensitivity further amplified by load-balancing constraints imposed during MoE pretraining. Because fine-tuned experts exhibit distinct specializations, even modest misrouting can cause severe performance degradation. To address this issue, we propose Hessian-Aware Router Calibration (HARC), a training-free framework that leverages second-order curvature information to realign the merged router. This approach admits a closed-form solution that can be efficiently solved using a matrix-free conjugate gradient method. Experiments on mathematical reasoning and code generation tasks show that HARC effectively mitigates routing breakdown across diverse MoE merging baselines and leads to substantial performance improvements.

Building powerful AI models is expensive. A popular cost-saving shortcut is “model merging”—combining several specialized AI models into a single, smarter one without the high cost of training from scratch. However, this shortcut fails when applied to a popular AI architecture called “Mixture-of-Experts” (MoE). We found that standard merging methods confuse the router, causing it to assign tasks to the wrong experts, which severely degrades the model’s performance. To fix this “routing breakdown,” we developed HARC (Hessian-Aware Router Calibration). HARC is a training-free method that recalibrates the router after merging, ensuring tasks are directed to the correct experts. Unlike other solutions, it requires no expensive retraining. In experiments on mathematical reasoning and coding tasks, HARC successfully fixed the routing errors and significantly boosted the merged model’s performance. Our work makes it possible to reliably combine these complex AI models, saving both computational resources and energy.