On the Emergence of Implicit Curriculum in RLVR Learning Dynamics

Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theory of the training dynamics of RLVR for transformers on compositional reasoning tasks. Our theory shows that mixed-difficulty training naturally induces an implicit curriculum: without any explicit schedule, easier problems become learnable first and shape the frontier for harder ones, creating a learning progression from easy to hard during optimization. The effectiveness of this curriculum is governed by the smoothness of the difficulty spectrum. When the spectrum is smooth, training dynamics enter a well-behaved relay regime, in which persistent gradient signals on easier problems make slightly harder ones tractable and keep training at the edge of competence. When the spectrum contains abrupt discontinuities, training undergoes grokking-type phase transitions with prolonged plateaus before progress recurs. As a technical contribution, our analysis develops and adapts techniques from Fourier analysis on finite groups to our setting. We validate the predicted mechanisms empirically via synthetic experiments.

A central puzzle in modern reasoning models is how they learn long chains of reasoning when training rewards only the final answer. We show that the answer lies in the structure of the training problems. When the training set contains a smooth range of difficulties, outcome-based rewards induce an implicit curriculum. The model learns the easiest cases first, and the training signal from those cases makes nearby harder cases reachable. Learning then advances like a relay: each newly mastered difficulty level passes useful signal to the next. This keeps training near the model’s current boundary of competence, even without any hand-designed schedule. The key determinant is the difficulty spectrum. Smooth spectra lead to steady progress, while sharp gaps can produce long plateaus followed by sudden improvement. Our theory explains when outcome-based reinforcement learning can gradually extend reasoning ability, and when it is likely to progress through abrupt phase transitions.