Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure.

Large language models are expected to perform well on many tasks, such as reasoning, instruction following, and multilingual understanding. A practical way to make such a model is "model merging": combining several single-task models, each trained for a different task, into one model. However, if all models are combined only once at the end, the final model may lose some abilities because knowledge from different tasks can conflict. In this paper, we propose a more gradual approach called a "many-shot merging" framework. Instead of combining all single-task models at once, we combine them step by step, allowing the final model to keep knowledge from different tasks more reliably. Based on this idea, we propose METIS, which gives more attention to tasks that have lost performance and carefully decides which changes should be included. Our results show that METIS improves the final model across many tasks while also helping the worst-performing task recover from performance loss. This shows that combining models gradually and carefully can lead to a stronger and more balanced language model.