Quantifying Cross-Domain Knowledge Distillation in the Presence of Domain Shift

This paper presents a theoretical investigation into the generalization capabilities of cross-domain knowledge distillation. Utilizing a high-dimensional asymptotic analysis of a linear teacher–student model, we characterize the excess risk while accounting for both model and covariate shifts. Our results provide a formal guarantee for the efficacy of distillation: even when the source and target domains differ substantially, there still may exist a regime where the student model achieves superior generalization ability over the student-only baseline. Moreover, we identify a \textit{crossed double descent} phenomenon: the excess risk can vary non-monotonically with the teacher’s and student’s dimension-to-sample-size ratios. These results provide rigorous insight into when and why distillation helps across domains.

When training a machine learning model for a new task, we often lack enough labeled data. One solution is to use a "teacher" model trained on a related but different source domain to guide a "student" model on the target task—a technique called cross-domain knowledge distillation. For example, a voice assistant trained on standard American English might need to adapt to regional dialects. Intuitively, if the source and target domains are very different, this transfer might seem unhelpful or even harmful. Surprisingly, our mathematical analysis shows that even when domains differ substantially, distillation can still improve the student's performance compared to training it only on the limited target data. We also discovered a "crossed double descent" phenomenon, where the student's performance fluctuates in complex ways as the problem size and data availability change. These findings help explain when and why transferring knowledge across different domains works, offering practical guidance for improving model training in data-scarce scenarios.