Timezone: »
Poster
Student Specialization in Deep Rectified Networks With Finite Width and Input Dimension
Yuandong Tian
Tue Jul 14 07:00 AM  07:45 AM & Tue Jul 14 08:00 PM  08:45 PM (PDT) @ None #None
We consider a deep ReLU / Leaky ReLU student network trained from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). The student network is \emph{overrealized}: at each layer $l$, the number $n_l$ of student nodes is more than that ($m_l$) of teacher. Under mild conditions on dataset and teacher network, we prove that when the gradient is small at every data sample, each teacher node is \emph{specialized} by at least one student node \emph{at the lowest layer}. For twolayer network, such specialization can be achieved by training on any dataset of \emph{polynomial} size $\mathcal{O}( K^{5/2} d^3 \epsilon^{1})$. until the gradient magnitude drops to $\mathcal{O}(\epsilon/K^{3/2}\sqrt{d})$. Here $d$ is the input dimension, $K = m_1 + n_1$ is the total number of neurons in the lowest layer of teacher and student. Note that we require a specific form of data augmentation and the sample complexity includes the additional data generated from augmentation. To our best knowledge, we are the first to give polynomial sample complexity for student specialization of training twolayer (Leaky) ReLU networks with finite depth and width in teacherstudent setting, and finite complexity for the lowest layer specialization in multilayer case, without parametric assumption of the input (like Gaussian). Our theory suggests that teacher nodes with large fanout weights get specialized first when the gradient is still large, while others are specialized with small gradient, which suggests inductive bias in training. This shapes the stage of training as empirically observed in multiple previous works. Experiments on synthetic and CIFAR10 verify our findings. The code is released in \url{https://github.com/facebookresearch/luckmatters}.
Author Information
Yuandong Tian (Facebook AI Research)
More from the Same Authors

2019 Poster: ELF OpenGo: an analysis and open reimplementation of AlphaZero »
Yuandong Tian · Jerry Ma · Qucheng Gong · Shubho Sengupta · Zhuoyuan Chen · James Pinkerton · Larry Zitnick 
2019 Oral: ELF OpenGo: an analysis and open reimplementation of AlphaZero »
Yuandong Tian · Jerry Ma · Qucheng Gong · Shubho Sengupta · Zhuoyuan Chen · James Pinkerton · Larry Zitnick 
2018 Poster: Gradient Descent Learns Onehiddenlayer CNN: Don't be Afraid of Spurious Local Minima »
Simon Du · Jason Lee · Yuandong Tian · Aarti Singh · Barnabás Póczos 
2018 Oral: Gradient Descent Learns Onehiddenlayer CNN: Don't be Afraid of Spurious Local Minima »
Simon Du · Jason Lee · Yuandong Tian · Aarti Singh · Barnabás Póczos 
2017 Poster: An Analytical Formula of Population Gradient for twolayered ReLU network and its Applications in Convergence and Critical Point Analysis »
Yuandong Tian 
2017 Talk: An Analytical Formula of Population Gradient for twolayered ReLU network and its Applications in Convergence and Critical Point Analysis »
Yuandong Tian