Self-Distillation for Data-Scarce Language Model Pretraining

Language model training is increasingly bottlenecked by data. Naive multi-epoch training quickly overfits, which motivates the search for effective regularization under data scarcity. In this work, we study the effectiveness of self-distillation as a regularization method. Considering eight scarcity levels and two model scales, we find that self-distillation serves as effective regularization when data are scarcer than what Chinchilla prescribes, outperforming both direct training and common regularization methods such as weight decay and exponential moving average. For practical recommendations on self-distillation, we suggest using larger models which are more data efficient despite the greater potential to overfit, and using plain teacher logits without temperature scaling.