InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Upweighting high-quality data in LLM pretraining often improves performance, but in data-limited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce \textbf{InfoLaw} (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scale-dependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger-scale runs (up to 7B, 425B tokens) with 0.15\% mean and 0.96\% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.

Large language models learn from enormous text collections, but the best training text is limited. Reusing the same high-quality text too many times can make training less efficient, while adding more varied but lower-quality text makes it hard to know the right balance. Existing scaling laws usually predict improvement from model size and compute, but they do not reliably account for data quality or repeated data. We introduce InfoLaw, a method for predicting how well a language model will perform under different training-data mixtures. InfoLaw treats training as a process of accumulating useful information: higher-quality data provides more value, while repeated exposure to the same data gives diminishing returns. Using smaller training runs, InfoLaw can predict the performance of larger models and new data recipes without testing every option directly. This helps researchers choose better training data mixtures with fewer costly experiments, especially when high-quality data is scarce. More broadly, it provides a practical way to train large language models more efficiently under real-world data limits.