An Information-Theoretic Criterion for Efficient Data Synthesis

Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent. We provide an information-theoretic account of this inconsistency: synthetic data improves a model only when the generation-training loop is information-open, i.e., shaped by external signals (verifiers, environments, or rubrics) that inject task-relevant information beyond the model's current distribution. When the loop is information-closed (relying on the model's own outputs without such signals), the data processing inequality ensures that task-relevant information can only decrease, making collapse a predicted outcome. Among information-open pipelines, both efficiency and generalization hinge on the meta-level of supervision: a coarser signal such as binary correctness treats all acceptable outputs as equivalent, so the behavior it teaches is not tied to any particular domain or surface form and generalizes naturally across tasks and domains. These observations lead to a guiding thesis: learning preferentially converges to the most information-efficient signal component available, which accelerates learning when that component is the intended one, but causes reward hacking when a spurious pattern happens to be simpler.

AI models are increasingly trained on data generated by other AI systems, a practice known as synthetic data. This approach has led to breakthroughs in mathematics and coding, but can also cause models to progressively lose their abilities — a phenomenon called model collapse. Why does synthetic data sometimes help and sometimes hurt? We use information theory to answer this question. We find that synthetic data only helps when the training process includes an independent judge — such as a math verifier or a code tester — that provides feedback the model cannot produce on its own. Without such a judge, the model simply recycles its own outputs, and each round of training loses a little information, leading to gradual degradation. We also discover that the most effective feedback is broad and simple. A judge that only says "right or wrong" teaches more efficiently than one that prescribes a specific answer, because each piece of feedback applies across many different situations. However, this same principle means that if a misleading shortcut happens to be simpler than the intended signal, models will learn the shortcut first — explaining a well-known failure mode called reward hacking.