Language Generation with Replay: A Learning-Theoretic View of Model Collapse

As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed. At the same time, widespread LLM usage increases the volume of machine-generated content on the web; together, these trends raise the likelihood of generated text re-entering future training corpora, increasing the associated risk of performance degradation often called model collapse. In practice, model developers address this concern through data cleaning, watermarking, synthetic-data policies, or, in some cases, blissful ignorance. However, the problem of model collapse in generative models has not been examined from a learning-theoretic perspective: we study it through the theoretical lens of the language generation in the limit framework, introducing a replay adversary that augments the example stream with the generator's own past outputs. Our main contribution is a fine-grained learning-theoretic characterization of when replay fundamentally limits generation: while replay is benign for the strongest notion of uniform generation, it provably creates separations for the weaker notions of non-uniform generation and generation in the limit. Interestingly, our positive results mirror heuristics widely used in practice, such as data cleaning, watermarking, and output filtering, while our separations show when these ideas can fail.

Large language models like ChatGPT are trained on web data, but the web now contains a growing amount of text produced by language models themselves. This creates a potentially harmful feedback loop: instead of learning from humans, new models may learn from the outputs of earlier models. Over time, this can cause their performance to collapse. Large language models are very complex, which makes this feedback loop hard to theoretically analyze in a direct way. We study it using a simple abstract game between a generator and an adversary. The generator tries to learn how to speak a language by observing examples chosen by the adversary. At each round of the game, the adversary can either show a correct sentence or replay a sentence previously produced by the generator, which may mislead it. Our results show that this feedback loop does not always make generation harder, but when it does, the barrier can be fundamental. They also help explain why mitigation strategies, such as filtering the inputs and outputs of these models, can work in some cases, while showing that such strategies cannot succeed in full generality.