THE CONFERENCE
FOR AUTHORS
FOR PARTICIPANTS

Timezone: »

 
Poster
The Shattered Gradients Problem: If resnets are the answer, then what is the question?
David Balduzzi · Marcus Frean · Wan-Duo Ma · Brian McWilliams · Lennox Leary · John Lewis

Tue Aug 08 01:30 AM -- 05:00 AM (PDT) @ Gallery #5
in Posters Tue »

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new ``looks linear'' (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections.

Author Information

David Balduzzi (Victoria University Wellington)
Marcus Frean (Victoria University Wellington)
Wan-Duo Ma (Victoria University of Wellington)
Brian McWilliams (Disney Research)
Lennox Leary (VUW)
John Lewis (Frostbite Labs and Victoria University)

J.P.Lewis is a numerical programmer and researcher working in computer graphics and computer vision. He has received credits on a few movies including Avatar and The Matrix Sequels, and several of his algorithms have been adopted in commercial software (Maya, Matlab). He has also worked in academic research, most recently with Victoria University in New Zealand. Lewis is currently Principal Technical Director at Frostbite Labs, Electronic Arts.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors