Timezone: »

 
Poster
Comparing Dynamics: Deep Neural Networks versus Glassy Systems
Marco Baity-Jesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli

Wed Jul 11 09:15 AM -- 12:00 PM (PDT) @ Hall B #168

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are the complexity of the loss-landscape and of the dynamics within it, and to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and data-sets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, thus showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Author Information

Marco Baity-Jesi (Columbia University)
Levent Sagun (ENS/CEA)
Mario Geiger (EPFL)
Stefano Spigler (EPFL)
Gerard Arous
Chiara Cammarota (King's College London)

Sep 2015 - present Lecturer in the Mathematics Department, King's College London Apr 2013 - Aug 2015 Researcher in the Physics Department, Sapienza University of Rome Nov 2009 - Mar 2013 Post-Doc in the Institut de Physique Theorique, CEA, Saclay Nov 2006 - Oct 2009 PhD student in the Physics Department, Sapienza University of Rome Nov 2004 - Oct 2006 Master Degree in Theoretical Phisics in the Physics Department, Sapienza University of Rome Sep 2001 - Nov 2004 Undergraduate studies in the Physics Department, Sapienza University of Rome

Yann LeCun (New York University)
Matthieu Wyart
Giulio Biroli

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors