Timezone: »

 
Poster
Go Wide, Then Narrow: Efficient Training of Deep Thin Networks
Denny Zhou · Mao Ye · Chen Chen · Tianjian Meng · Mingxing Tan · Xiaodan Song · Quoc Le · Qiang Liu · Dale Schuurmans

Thu Jul 16 07:00 AM -- 07:45 AM & Thu Jul 16 06:00 PM -- 06:45 PM (PDT) @
in Poster Session 42 »

For deploying a deep learning model into production, it needs to be both accurate and compact to meet the latency and memory constraints. This usually results in a network that is deep (to ensure performance) and yet thin (to improve computational efficiency). In this paper, we propose an efficient method to train a deep thin network with a theoretic guarantee. Our method is motivated by model compression. It consists of three stages. In the first stage, we sufficiently widen the deep thin network and train it until convergence. In the second stage, we use this well-trained deep wide network to warm up (or initialize) the original deep thin network. This is achieved by letting the thin network imitate the immediate outputs of the wide network from layer to layer. In the last stage, we further fine tune this well initialized deep thin network. The theoretical guarantee is established by using mean field analysis. It shows the advantage of layerwise imitation over traditional training deep thin networks from scratch by backpropagation. We also conduct large-scale empirical experiments to validate our approach. By training with our method, ResNet50 can outperform ResNet101, and BERTBASE can be comparable with BERTLARGE, where both the latter models are trained via the standard training procedures as in the literature.

Keywords: Algorithms · Architectures · Deep Learning - Algorithms

Author Information

Denny Zhou (Google Brain)
Mao Ye (UT Austin)
Chen Chen (Google)
Tianjian Meng (Google Brain)
Mingxing Tan (Google Brain)
Xiaodan Song (Google Brain)
Quoc Le (Google Brain)
Qiang Liu (UT Austin)
Dale Schuurmans (Google / University of Alberta)

More from the Same Authors