Timezone: »

 
Spotlight
Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective
Jingzhao Zhang · Haochuan Li · Suvrit Sra · Ali Jadbabaie

Tue Jul 19 01:15 PM -- 01:20 PM (PDT) @ Room 309
in OPT: Non-Convex »

This work examines the deep disconnect between existing theoretical analyses of gradient-based algorithms and the practice of training deep neural networks. Specifically, we provide numerical evidence that in large-scale neural network training (e.g., ImageNet + ResNet101, and WT103 + TransformerXL models), the neural network's weights do not converge to stationary points where the gradient of the loss is zero. Remarkably, however, we observe that even though the weights do not converge to stationary points, the progress in minimizing the loss function halts and training loss stabilizes. Inspired by this observation, we propose a new perspective based on ergodic theory of dynamical systems to explain it. Rather than studying the evolution of weights, we study the evolution of the distribution of weights. We prove convergence of the distribution of weights to an approximate invariant measure, thereby explaining how the training loss can stabilize without weights necessarily converging to stationary points. We further discuss how this perspective can better align optimization theory with empirical observations in machine learning practice.

Keywords: DL: Theory · OPT: Non-Convex

Author Information

Jingzhao Zhang (Tsinghua University)
Haochuan Li (MIT)
Suvrit Sra (MIT & Macro-Eyes)
Ali Jadbabaie (Massachusetts Institute of Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors