What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization. While existing studies offer isolated insights, a unified theoretical framework explaining the efficacy of KT across these disparate regimes remains lacking. In this work, we establish a unified spectral analysis of SGD dynamics in high-dimensional linear regression, elucidating the efficiency of KT across seemingly disparate regimes. We characterize KT efficiency through two distinct mechanisms: \emph{Spectral Horizon Expansion} in KD, which enables the capture of statistically inaccessible high-frequency signals, and \emph{Spectral Denoising} in W2S, where the student acts as a filter for optimization noise. Our framework unifies these phenomena, revealing that the efficacy of transfer is governed by the interplay between implicit regularization and heterogeneous spectral learning speeds over the spectrum.

In modern artificial intelligence, having a "student" model learn from a "teacher" model—a process known as Knowledge Transfer—is a cornerstone technique. This includes training small, efficient models to mimic large ones (Knowledge Distillation), as well as a surprising new trend where a highly capable student learns from a much weaker teacher and somehow surpasses it (Weak-to-Strong Generalization). Despite widespread empirical success, a unified theory explaining exactly why models learn better from each other's predictions than from raw data has been missing. This paper introduces a comprehensive framework that uncovers the hidden mechanics behind these phenomena. We demonstrate that knowledge transfer succeeds through two distinct mechanisms: * The Teacher as a Guide: When a strong teacher instructs a weaker student, the teacher expands the student's learning horizon, helping it capture complex details in the data that the student could never discover on its own. * The Student as a Filter: When a strong student learns from a weak teacher, the student acts as an intelligent filter. It successfully tunes out the weak teacher's random errors and optimization noise, extracting only the true underlying signals to ultimately outperform its instructor. Ultimately, this research demystifies how AI systems interact during supervision and provides clear, theoretical criteria for what truly makes an AI model "strong".