The Power of Power Law: Asymmetry Enables Compositional Reasoning

Natural language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency. While a common intuition suggests that reweighting or curating data toward a uniform distribution may help models better learn these long-tail skills, we find a counterintuitive result: across a wide range of compositional reasoning tasks, such as state tracking and multi-step arithmetic, training under power-law distributions consistently outperforms training under uniform distributions. To understand this advantage, we introduce a minimalist skill-composition task and show that learning under a power-law distribution provably requires significantly less training data. Our theoretical analysis reveals that power law sampling induces a beneficial asymmetry that improves the pathological loss landscape, which enables models to first acquire high-frequency skill compositions with low data complexity, which in turn serves as a stepping stone to efficiently learn rare long-tailed skills. Our results offer an alternative perspective on what constitutes an effective data distribution for training models.

AI models are trained on text where some topics appear frequently and others rarely, following a power-law distribution. A natural way is to balance this data so that rare topics get more attention during training. But is that actually the right approach? We tested this across reasoning tasks like multi-step math problems and state tracking, and found a surprising result: training on the natural, unbalanced distribution consistently outperformed training on balanced data. We also proved theoretically that power-law training requires substantially less data to achieve the same performance on a toy task. The key insight is that frequent patterns act as building blocks. When a model first masters common skills, it gains a foundation for learning rare ones more efficiently.