Model Merging Scaling Laws in Large Language Models

We study empirical scaling laws for language model merging measured by cross-entropy. Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we add experts or scale the model size. We identify a compact power law that links model size and expert number: the size-dependent floor decreases with model capacity, while the merging tail exhibits clear diminishing returns in the number of experts. The law holds in-domain and cross-domain, tightly fits measured curves across diverse architectures and methods (Average, TA, TIES, DARE), and explains two robust regularities: most gains arrive early, and variability shrinks as more experts are included. Building on this, we present a simple theory that explains why gains fall roughly as (1/k) and links the floor and tail to properties of the base model and the diversity across domains. This law enables \emph{predictive planning}: estimate how many experts are needed to reach a target loss, decide when to stop adding experts, and trade off scaling the base model versus adding experts under a fixed budget—turning merging from heuristic practice into a computationally efficient, planable alternative to multitask training. This suggests a scaling principle for distributed generative AI: predictable gains can be achieved by composing specialists, offering a complementary path toward AGI-level systems.

Large language models are often adapted to different tasks or domains, creating many specialized models. A practical question is whether we can combine these specialists into one model without retraining it from scratch. This process, called model merging, is attractive because it can save substantial computation, but until now it has been hard to predict how much benefit we will get by merging more models or by starting from a larger base model. In this work, we study thousands of merged language models and find a simple pattern: most of the improvement comes from the first few specialists, and the benefit of adding more specialists gradually decreases. We also find that larger base models are easier to merge and that different merging methods become more similar as the scale grows. Based on these observations, we propose a compact rule that predicts the loss of a merged model from two quantities: the size of the base model and the number of specialists being merged. This rule can help practitioners decide how many models are worth merging, when to stop adding more specialists, and whether to spend a fixed budget on larger base models or more experts. Overall, our results turn model merging from a mostly trial-and-error procedure into a more predictable and budget-aware way to build capable language models.