Timezone: »

 
Poster
The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
Karthik Abinav Sankararaman · Soham De · Zheng Xu · W. Ronny Huang · Tom Goldstein

Wed Jul 15 10:00 AM -- 10:45 AM & Wed Jul 15 11:00 PM -- 11:45 PM (PDT) @
in Poster Session 24 »

This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But when gradient confusion is low, data samples interact harmoniously, and training proceeds quickly. Through theoretical and experimental results, we demonstrate how the neural network architecture affects gradient confusion, and thus the efficiency of training. Our results show that, for popular initialization techniques, increasing the width of neural networks leads to lower gradient confusion, and thus faster model training. On the other hand, increasing the depth of neural networks has the opposite effect. Our results indicate that alternate initialization techniques or networks using both batch normalization and skip connections help reduce the training burden of very deep networks.

Keywords: Deep Learning Theory · Deep Learning - Theory
[ Slides] 

Author Information

Karthik Abinav Sankararaman (Facebook)
Soham De (DeepMind)
Zheng Xu (Google Research)
W. Ronny Huang (Google AI)
Tom Goldstein (University of Maryland)

More from the Same Authors