Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

This work investigates the "small-vs-large gap", where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.

Training AI systems usually improves when they see more examples. Surprisingly, prior work has observed a surprising training effect: for some reasoning-like tasks, training on fewer examples while repeating them more often can require less computation than training on a larger dataset. We study this "small-vs-large gap" more systematically. We confirm that the effect appears across different algorithmic tasks, model architectures, and optimization methods. More importantly, we provide an explanation for why it happens. Our analysis suggests that the sampling biases of smaller datasets can help different layers of a model grow at better relative speeds during training. This layer-wise growth makes optimization more efficient. Our results show that repeating a smaller dataset is not merely a backup strategy when data is limited. Instead, carefully using fewer examples with more repetition may be a practical way to guide learning, especially for tasks that require reasoning.