Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models

Large Language Models (LLMs) are pretrained on massive datasets and later instruction-tuned via supervised fine-tuning (SFT) or reinforcement learning (RL). Best practices emphasize large, diverse pretraining data, whereas post-training operates differently: SFT relies on smaller, high-quality datasets, while RL benefits more from scale, with larger amounts of feedback often outweighing label quality. Yet it remains unclear why pretraining and RL require large datasets, why SFT excels on smaller ones, and what defines high-quality SFT data. In this work, we theoretically analyze transformers trained on an in-context weight prediction task for linear regression. Our analysis reveals several key findings: $(i)$ balanced pretraining data can induce latent capabilities later activated during post-training, and $(ii)$ SFT learns best from a small set of examples challenging for the pretrained model, while excessively large SFT datasets may dilute informative pretraining signals. In contrast, RL is most effective on large-scale data that is not overly difficult for the pretrained model. We validate these theoretical insights with experiments on large nonlinear transformer architectures.

Large Language Models are trained in two stages. First, they learn general patterns from vast amounts of text (pretraining). Then, they are refined using smaller datasets to improve specific behaviors (post-training). Although practitioners know that these stages require very different kinds of data, it has not been well understood why. In this work, we develop a mathematical framework to study how these stages interact. We find that diverse and well-balanced pretraining data helps the model develop “hidden” abilities that are not immediately visible but can be activated later during post-training. We also explain why different post-training methods behave so differently. Supervised fine-tuning works best when using a small number of carefully chosen, challenging examples—too much data can actually weaken the useful structure learned during pretraining. In contrast, reinforcement learning improves steadily with larger amounts of data, as long as the tasks are not too difficult for the model to learn from. These results provide a clearer understanding of how to design training data for modern AI systems, helping make them more efficient, reliable, and effective.