Explaining Data Mixing Scaling Laws

Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures. However, a theoretical understanding of these model loss behaviors remains absent. In this work, we propose a unified framework to explain the underlying mechanics of data mixing. Our approach extends theoretical perspectives originally developed for standard neural scaling laws (e.g., Kaplan and Chinchilla) to the multi-domain setting. Based on the distributional assumption that domains overlap on fundamental skills while diverging on specialized skills, we identify two key factors that govern the domain losses of models trained on different data mixtures: (1) Capacity Competition, where the allocation of finite model capacity couples domain losses globally, and (2) Noise Reduction, where optimal weights shift toward harder-to-learn domains to minimize overall noise. Empirical evaluations show that our framework outperforms existing baselines by fitting the loss landscape with a lower Mean Relative Error and identifying higher-performing training mixtures. Most importantly, our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales using parameters fitted on smaller ones. In addition, our model achieves these results using significantly fewer parameters compared to previous empirical laws.

Large foundation models are trained on massive datasets spanning multiple domains, such as math, code, and literature. The precise proportion of each domain—the "data mixture"—drastically impacts the final capabilities of the model. However, finding the optimal mixture traditionally relies on computationally expensive, trial-and-error guesswork. To avoid this guesswork, researchers use empirical "scaling laws" to predict model performance before training begins. Yet, these current formulas act as black boxes: they successfully fit past data, but they lack a theoretical explanation for why different data domains interact the way they do. This paper introduces a unified theoretical framework to explain these hidden mechanics. The framework is built on a natural assumption: different data domains share fundamental, basic skills, but possess entirely unique, specialized skills. Based on this, the authors identify two core forces that drive how an AI learns: (1) Capacity Competition: A model has a finite capacity to learn. The highly specialized skills from different domains are in competition for this limited space. Adjusting the data mixture changes how the AI prioritizes and allocates its capacity across different subjects. (2) Noise Reduction: For the AI to master a skill, it needs repeated exposure. Because some subjects are inherently more difficult to learn than others, the optimal data mixture must shift weights toward these complex domains to minimize errors (or "noise") in the final model.