Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning

Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential -- yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to $2.84\%$ and $5.47\%$ on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to $98.7\%$ over the traditional exhaustive approach.

Training large AI language models is extremely expensive, often requiring hundreds of individual training runs to understand how performance scales with increasing computational resources. This knowledge, captured in so-called scaling laws, is invaluable for making informed decisions about how to train and deploy these models efficiently. Yet the process of estimating these scaling laws has itself remained surprisingly wasteful: researchers typically train every candidate model until completion, even when many of those runs contribute little useful information. We address this by asking a simple but underexplored question: can we strategically decide which models to train, and for how long, to obtain accurate scaling laws at a fraction of the usual cost? We adapt a resource allocation technique called Successive Halving and augment it with probabilistic surrogate models that predict how a model's performance will evolve before it has finished training. This allows us to identify and invest compute budget in the most promising models early, while stopping less useful ones sooner. Our experiments show that this approach recovers accurate scaling laws while reducing computational costs by up to 98.7%, making scaling law research more accessible to the broader research community, including institutions without access to the massive compute budgets currently required by state-of-the-art approaches.