Why Self-Distillation Helps and Hurts: Denoising vs. Signal Forgetting

Self-distillation, a special case of self-training, repeatedly trains a model on pseudo-labels produced by its previous iterate. Why can this iteration improve generalization, and why can it later hurt? We answer this question in overparameterized linear regression, starting from an estimator fit to noisy labels and then repeatedly refitting on fresh covariates labeled by the current estimator. In the proportional asymptotic regime, we derive deterministic recursions for the test risk and prove concentration around them. The recursions reveal a simple trade-off: iteration \emph{denoises} by attenuating stochastic error inherited from the initial fit, but it also induces \emph{signal forgetting} because each teacher--student transfer projects the estimate through a new data-dependent subspace. This competition produces a $U$-shaped risk curve and an optimal stopping time. The same dynamics act as a spectral filter: eigendirections carrying more of the true regression signal are retained longer, while weakly aligned or noise-like directions are suppressed. We further develop an iterated generalized cross-validation criterion with uniform consistency for selecting the stopping time. Synthetic simulations corroborate the recursions and data-driven stopping rule, while neural-network experiments exhibit the same denoising--forgetting pattern.

Many modern AI systems improve themselves by training on labels produced by an earlier version of the same model. This process, often called self-distillation or self-training, can help, but it can also make the model worse if repeated too many times. Our paper explains why both effects happen. We study a clean mathematical model of this process. The first model is trained on noisy labels. Later models are trained on new unlabeled examples, using predictions from the previous model as pseudo-labels. We show that iteration creates a trade-off. Early rounds can remove noise inherited from the initial model, improving prediction. But each teacher-to-student transfer also loses some information about the true signal, so later rounds can cause performance to deteriorate. This produces a $U$-shaped error curve and suggests that self-distillation should be stopped at the right time. We also develop a data-driven method, based on iterated generalized cross-validation, for selecting this stopping time. Simulations and neural-network experiments support the same denoising-versus-forgetting picture.