Timezone: »

 
Knowledge Distillation for Efficient Sequences of Training Runs
Xingyu Liu · Xingyu Liu · Alexander Leonardi · Alexander Leonardi · Lu Yu · Lu Yu · Christopher Gilmer-Hill · Christopher Gilmer-Hill · Matthew Leavitt · Matthew Leavitt · Jonathan Frankle · Jonathan Frankle

@
in The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward »
Event URL: https://openreview.net/forum?id=kksQ0J87f03 »

In many practical scenarios - like hyperparameter search or continual retraining with new data - related training runs are performed many times in sequence. Current practice is to train each of these models independently from scratch. We study the problem of exploiting the computation invested in previous runs to reduce the cost of future runs using knowledge distillation (KD). We find that augmenting future runs with KD from previous runs dramatically reduces the time necessary to train these models, even taking into account the overhead of KD. We improve on these results with two strategies that reduce the overhead of KD by 80-90% with minimal effect on accuracy and vast pareto-improvements in overall cost. We conclude that KD is a promising avenue for reducing the cost of the expensive preparatory work that precedes training final models in practice.

Author Information

Xingyu Liu (Harvard University, Harvard University)
Xingyu Liu (Harvard University, Harvard University)
Alexander Leonardi (Harvard University)
Alexander Leonardi (Harvard University)
Lu Yu (Harvard University, Harvard University)
Lu Yu (Harvard University, Harvard University)
Christopher Gilmer-Hill (Harvard University)
Christopher Gilmer-Hill (Harvard University)
Matthew Leavitt (Facebook AI Research (FAIR))
Matthew Leavitt (Facebook AI Research (FAIR))
Jonathan Frankle (MosaicML / Harvard)
Jonathan Frankle (MosaicML / Harvard)

More from the Same Authors