Poster

Optimization Theory for ReLU Neural Networks Trained with Normalization Layers

Yonatan Dukler · Quanquan Gu · Guido Montufar

Keywords: [ Optimization ] [ Deep Learning Theory ] [ Non-convex Optimization ] [ Deep Learning - Theory ]

[ Abstract ]
Wed 15 Jul 9 a.m. PDT — 9:45 a.m. PDT
Wed 15 Jul 8 p.m. PDT — 8:45 p.m. PDT

Abstract:

The current paradigm of deep neural networks has been successful in part due to the use of normalization layers. Normalization layers like Batch Normalization, Layer Normalization and Weight Normalization are ubiquitous in practice as they improve the generalization performance and training speed of neural networks significantly. Nonetheless, the vast majority of current deep learning theory and non-convex optimization literature focuses on the un-normalized setting. We bridge this gap by providing the first global convergence result for 2 layer non-linear neural networks with ReLU activations trained with a normalization layer, namely Weight Normalization. The analysis shows how the introduction of normalization layers changes the optimization landscape and in some settings enables faster convergence as compared with un-normalized neural networks.

Chat is not available.