Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question:Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data.

LLMs improve at reasoning by attempting problems, and reinforcing correct answers. This method has an important limitation: models cannot learn on problems where they have a zero percent success rate, because these problems provide no learning signal. Generalizing to these hard problems often requires a human-curated curriculum of easier problems that act as "stepping stones". Our paper shows that models can learn to generate a curriculum for themselves to solve hard math problems. In our framework, we start with two copies of the same model: a teacher, and a student. The teacher generates candidate problems that the student learns to solve. The teacher gets rewarded only if the student improves on the target set of hard problems, which forces the teacher to produce useful questions. With our framework, we find that models can teach themselves to get "unstuck" on problems that they initially failed at. Our results show that the ability to generate a useful curriculum for hard problems is separate from the ability to solve those problems. Our work suggests a principled path for AI systems to expand their own capabilities beyond what they can currently solve, without requiring additional human-curated data.