OCNR: Stabilizing Self-Play by Mitigating Iteration-Collapse With One-Class Novelty Rewards

Training large language models via self-play often suffers from a persistent iteration-collapse, where performance initially improves but subsequently regresses as training iterations increase. We analyze this phenomenon as arising from cross-iteration degeneration, where the task-generation distribution becomes increasingly confined to a narrow subset of familiar (seen) problems, weakening the effective learning signal and destabilizing training. To address this issue, we propose a plug-in approach that augments existing self-play pipelines with a one-class novelty reward. A Seen Detector trained on a historical buffer of previously used training problems identifies in-support instances and discourages redundant generation by the questioner, thereby steering exploration toward under-explored yet learnable regions. Experimental results show that the proposed method mitigates iteration-collapse during iterative training and yields consistent improvements.

Large language models can improve their reasoning abilities through self-play, where one part of the model generates problems and another part solves them, forming a self-improving loop without human supervision. In practice, however, this loop often suffers from iteration-collapse: performance initially improves but then degrades as training continues. We found that a key driver of this collapse is the problem generator drifting toward repetitive, already-learned problems across iterations, even when existing methods try to encourage surface-level variety in generation. The core issue is not just diversity but novelty: problems must be genuinely new from the solver's perspective. To address this, we built a lightweight "novelty detector" that tracks what the solver has already trained on in its own representation space. When the generator proposes a problem too close to previously seen ones, it receives a penalty that steers it toward unexplored yet solvable regions. In experiments on math and coding benchmarks, our method prevents iteration-collapse and enables models to keep improving over many more training rounds.