Improved Scaling Laws via Weak-to-Strong Generalization in Random Features Ridge Regression

It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random features ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher---in fact, when the test error of the teacher does not even decay with the sample size.

We analyze a simplified mathematical model in which both the weak teacher and the strong student learn from random features. This setting allows us to precisely describe how the student’s test error changes as the amount of data grows. Our analysis shows that the student can improve not only by reducing systematic mistakes, but also by reducing errors caused by randomness in the training data. The main takeaway is that weak supervision can be much more powerful than it may first appear. In some regimes, the strong student reaches the best possible learning rate even when the weak teacher’s own error does not improve with more data. This helps explain why training stronger models from weaker model-generated labels can succeed, and clarifies when this strategy is likely to be effective.