On the Generalization Gap in Self-Evolving Language Model Reasoning

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the SE algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework, including single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that SE consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non trivial gap to oracle supervision. We find that multi-turn critic-revision with large models could reach strong self-evolution performance, where Gemma 12B nearly matches oracle-supervised training. Beyond KK, we also evaluate SE on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop SE can help, and show how internally generated supervision remains insufficient under this minimal formulation.

Many researchers have been exploring whether AI systems can improve themselves. This is potentially possible if the AI system can look at a bunch of questions, generate candidate answers, and then learn from how it solves (or fails to solve) the questions. However, we find this is pretty tough to do. We compare two settings: (1) the model generates data to train itself versus (2) the model uses human-labeled or expert-written examples to train. In almost all cases, the second approach is significantly better, so (2) is better than (1). There are some cases where the model can become more confident at solving certain type of problems by itself. But, in general, we systematically show that it is much better to train with high-quality data than to have models teach themselves new things.