On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Recent reinforcement learning (RL) techniques have yielded impressive reasoning improvements in language models, yet it remains unclear whether RL truly extends a model's reasoning ability beyond pre-training. A central challenge is the lack of control in modern training pipelines, where opaque pre-training data, underexplored mid-training, and complex RL interactions obscure causal effects. To resolve this ambiguity, we develop a controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training. Our approach employs synthetic reasoning tasks with explicit atomic operations, parseable step-by-step reasoning traces, and systematic manipulation of training distributions. We evaluate along: extrapolative generalization to more complex compositions and contextual generalization across surface contexts. Using this framework, we reconcile competing views on RL’s effectiveness: 1) RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and RL data target the model’s edge of competence, tasks that are difficult but not yet out of reach. 2) Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL reliably transfers. 3) Mid-training significantly enhances performance under fixed compute compared with RL alone, demonstrating its central but underexplored role. 4) Process-level rewards reduce reward hacking and improve reasoning fidelity. Together, these results clarify the interplay between pre-training, mid-training, and RL, offering a foundation for improving reasoning language models training strategies. Codes and data are avaialble at Github and HuggingFace

AI systems are now often asked to solve problems that require several steps of reasoning, such as math, coding, or scientific questions. But when these systems improve after being trained with feedback, it is hard to know what actually changed: did they learn a new way to reason, or did they simply become better at using abilities they already had? We study this question by building a controlled testbed where we can decide exactly what the model sees during each stage of training and what kind of reasoning each problem requires. This lets us compare three ingredients: broad early training, focused intermediate training, and later training from feedback. We find that feedback is most useful when problems are challenging but still within reach—not too easy, and not impossibly hard. We also find that a small amount of prior exposure to a new setting helps models transfer their reasoning there, and that focused intermediate training often complements feedback training under the same training budget. Finally, checking the model’s intermediate steps, rather than only its final answer, helps discourage shortcuts and leads to more trustworthy reasoning. Overall, our work offers practical guidance for training AI systems that reason more reliably.