How Recursive Training Collapses and What Can Be Done About It

When generative models are trained on data produced by earlier generative models, the quality of their outputs systematically degrades, a phenomenon known as \textit{model collapse}. But what exactly drives this collapse, how fast does it proceed, and can it be stopped? We show that two distinct mechanisms, drift in the estimated mean and contraction in the estimated variance, contribute \textit{equally} to the total information loss, resolving an ongoing debate in the literature. When variance is estimated from data (as all practical models do), the rate of degradation \textit{doubles} compared to the idealized case of known variance, making variance contraction a particularly dangerous ``silent killer'' that operates without obvious distributional shift. We then establish that each generation of recursive training acts as an optimal lossy compression step, and prove that no estimator can fundamentally outperform this rate: collapse is unavoidable without external intervention. For the common intervention strategy of mixing real data into each generation, we compare a constant vs a decaying mixing schedule and prove that gradually reducing the mixing fraction inevitably fails in the long run. Moreover, we show that under the Gaussian model, when total real data exposure is matched, constant mixing always outperforms decaying mixing schedules. Experiments on variational autoencoders, diffusion models, and GPT-2 validate the theoretical predictions.