OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall—LLM pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. It also outperforms previous data selection methods across different stages of training, including from-scratch pre-training and also mid-training. Beyond online selection, the OPUS utility score also demonstrates potential as a static filter for flagging and removing toxic documents from contaminated training corpora prior to training.

Training large language models requires enormous amounts of text, but the internet’s best material is running out — a challenge researchers call the “Data Wall.” Simply feeding models more random web pages no longer works; we need to choose which data actually helps the model learn at each step. Current approaches fall short. Static filters act like a fixed sieve applied before training, ignoring how the model’s needs change as it learns. Dynamic methods adjust during training, but they overlook a crucial detail: modern optimizers reshape how the model updates itself, so judging data by basic gradient signals misses the real picture. We built OPUS, a smart selector that picks training examples in real time by measuring how much each piece of text truly pushes the model toward better performance under the optimizer’s actual learning path. Using lightweight projection tricks and diversity-aware sampling, OPUS adds less than 5% extra computation while improving results across model sizes, optimizers, and training stages. As a bonus, the same scoring system can spot and remove toxic documents before training starts, acting like a security checkpoint for clean, efficient learning.