BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

Effective data selection is essential for pretraining large language models (LLMs), improving efficiency and generalization to downstream tasks. However, existing approaches often rely on external pretrained models, making it difficult to separate the benefits of data selection from those introduced by external models. In addition, many methods estimate data importance from a fixed model state or short-horizon update, making it hard to capture how data preference changes as the model evolves during pretraining. In this paper, we introduce BLISS (BileveL Influence Scoring method for data Selection), a lightweight data selection method that operates entirely \emph{from scratch}, without external pretrained oracle models, while modeling dynamic data preference. BLISS uses a small proxy model as a surrogate for the LLM and trains a score model to estimate sample importance through multi-step proxy updates induced by score-weighted training data. We formulate data selection as a bilevel optimization problem: the upper-level objective optimizes the score model to assign sample weights, so minimizing the lower-level weighted training loss improves validation performance. Once optimized, the score model predicts influence scores, enabling efficient selection of high-quality samples for LLM pretraining. We validate BLISS by pretraining 410M/1B/2.8B Pythia and LLaMA-0.5B models on selected C4 subsets. Under the 1B setting, BLISS achieves a $1.7\times$ speedup in reaching the same performance as the state-of-the-art method, while delivering superior performance across multiple downstream tasks.

Large language models are trained on massive text collections, but not all training examples are equally helpful. Choosing better data can make training faster and improve the final model. However, many existing methods use already-trained AI models to decide which data is useful, making it hard to separate the value of data selection from the value of those outside models. Other methods judge data using only a fixed moment in training, even though useful data may change as the model learns. We introduce BLISS, a method for selecting training data from scratch, without using external pretrained models. BLISS uses a small model to imitate how a larger model learns over multiple training steps, and learns to predict which examples will improve future performance. It then assigns scores to training examples so that high-quality data can be selected efficiently. In experiments on several language models trained with web text, BLISS reached the same performance faster than a strong existing method and achieved better results on multiple downstream tasks. This suggests that better data selection can reduce the cost of training large language models while improving their quality.