R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training

Self-play bootstraps LLM reasoning through an iterative Challenger--Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) Local Diversity Illusion, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) Surface Diversity Illusion, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods.

Modern AI language models can become smarter by inventing their own practice problems and learning from them — much like a student writing their own exercises to study. But this trick often stalls: after a few rounds the AI starts producing problems that look new on the surface yet secretly need the same reasoning to solve, so it keeps drilling the same trick and stops improving. We call this hidden trap the "diversity illusion". We built R-Diverse, a training recipe that gives the AI two new abilities. First, it keeps a long-term memory of every practice problem it has ever made and is penalized for inventing problems too similar to past ones, preventing silent recycling across rounds. Second, before deciding whether two problems are really different, it boils each one down to a small piece of code capturing the underlying solution recipe — and only treats them as different if their recipes differ. With these two pieces, AI models keep improving over many more rounds of self-practice instead of plateauing, making self-training a more reliable — and less human-annotation-hungry — route to better reasoning.