Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth labels and the teacher’s own predictions using the same architecture and training data. Although SD has been empirically shown to often improve generalization, its formal guarantees remain limited. We study SD for ridge regression with an unconstrained mixing weight $\xi \in \mathbb{R}$. Conditional on the training data and without any distributional assumptions, we prove that for any squared prediction risk $R$ (including out-of-distribution), the optimally mixed student strictly improves upon the ridge teacher at every regularization level $\lambda$ where the teacher risk is not stationary ($R'(\lambda) \neq 0$). We also characterize the optimal mixing weight $\xi^\star$ in terms of the risk derivative $R'$, showing that it can be negative, which is the case in over-regularized regimes. To quantify SD risk improvements, we derive exact risk asymptotics in the proportional asymptotics regime for general anisotropic covariance and deterministic signals. From a practical standpoint, we propose a consistent one-shot tuning method to estimate $\xi^\star$ without grid search, sample splitting, or refitting. Experiments on real-world datasets and pretrained neural network features support our theory and the one-shot tuning method.

Self-distillation is a process in which a model is retrained on the same data using a mix of its own predictions and the true labels. A mixing weight between 0 and 1 is commonly used to control how much the model learns from the true labels versus its own predictions. We study self-distillation in ridge regression where the mixing weight is allowed to take any real value, including negative ones, as recent studies have suggested that this can sometimes work better. Our first result gives a simple rule for determining when the retrained model (student) can outperform the original model (teacher) and when negative mixing is helpful. We also derive the exact performance of the student in large-scale settings where the number of data points and features grow together at the same rate. In some simple cases, the student can closely match the best possible model with extreme regularization. From a practical standpoint, we propose an efficient method for estimating the best mixing weight without testing many possible values or setting aside some data for tuning. Extensions such as performing distillation over multiple rounds, distillation using the teacher's labels on fresh data, or other ridge variants are also covered. Our results provide a simple and computationally cheap way to improve poorly tuned teachers, at least in ridge-type problems.