Feature Learning and Signal Propagation in Deep Neural Networks

Yizhang Lou · Chris Mingard · Soufiane Hayou

Hall E #306

Keywords: [ PM: Gaussian Processes ] [ PM: Spectral Methods ] [ T: Deep Learning ] [ T: Probabilistic Methods ] [ DL: Theory ]

[ Abstract ]
[ Poster [ Paper PDF
Wed 20 Jul 3:30 p.m. PDT — 5:30 p.m. PDT
Spotlight presentation: DL: Theory
Wed 20 Jul 7:30 a.m. PDT — 9 a.m. PDT


Recent work by Baratin et al. (2021) sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align much more with data compared to other layers (where the alignment is defined as the normalize euclidean product of the tangent features matrix and the data labels matrix). The curve of the alignment as a function of layer index (generally) exhibits a ascent-descent pattern where the maximum is reached for some hidden layer. In this work, we provide the first explanation for this phenomenon. We introduce the Equilibrium Hypothesis which connects this alignment pattern to signal propagation in deep neural networks. Our experiments demonstrate an excellent match with the theoretical predictions.

Chat is not available.