D²Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning

Reinforcement learning (RL) has demonstrated potential for enhancing reasoning in large language models (LLMs). However, effective RL training, which requires medium-difficulty training samples, faces two fundamental challenges: Effective Data Scarcity and Dynamic Difficulty Shifts, where medium-difficulty samples are scarce and become trivial as models improve. Existing methods mitigate this scarcity to some extent by generating training samples. However, these approaches suffer from anchor-free generation, ignoring co-evolution, and difficulty mismatch. To address these issues, we propose D²Evo, a Dual Difficulty-aware self-Evolution RL framework. In each iteration, our method mines medium-difficulty anchors based on the current Solver's capability, trains the Questioner to generate diverse questions at appropriate difficulty levels, and jointly optimizes both components to enable progressive reasoning gains. Extensive experiments demonstrate that D²Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples, and exhibits strong generalization on general reasoning benchmarks.

Teaching large language models (LLMs) to reason works best when practice problems match their current ability, neither too easy nor too hard. However, such appropriately challenging problems are scarce and become trivial as the model improves. We propose D2Evo, where two LLM roles co-evolve: a Questioner that generates new practice problems and a Solver that attempts them. At each stage, the system identifies appropriately challenging real problems, uses them as inspiration for the Questioner to create diverse new problems, and trains the Solver on this fresh sample. As the Solver improves, harder problems are promoted, keeping both roles at their learning frontier. Our experiments show D2Evo outperforms existing methods on math reasoning using fewer than 2,000 real problems, and generalizes to broader reasoning tasks.