When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75\%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate (~ 50%), while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling.

Modern AI models are often improved by making them larger and training them on more data, but high-quality text data is becoming limited. When the same data must be reused repeatedly, standard dense models can gain less from each additional training epoch. We study whether sparse training, where only selected parts of a model are active and updated, can make better use of limited data. Through extensive language-model training experiments, we measure how model size, data reuse, and sparsity interact, and we build a scaling rule that predicts this behavior. We find that moderate sparsity helps models benefit from repeated data for longer instead of saturating quickly. It also changes the best way to spend computation: sparse models often benefit more from training longer than from simply adding more active parameters. In our experiments, sparse models can match dense-model accuracy while using much less training computation at the best sparsity level. These results suggest that sparsity is not only an efficiency trick, but also a practical way to train capable language models when high-quality data is scarce.