Timezone: »

Understanding the Dynamics of Gradient Flow in Overparameterized Linear models
Salma Tarmoun · Guilherme Franca · Benjamin Haeffele · Rene Vidal

Wed Jul 21 05:45 PM -- 05:50 PM (PDT) @

We provide a detailed analysis of the dynamics ofthe gradient flow in overparameterized two-layerlinear models. A particularly interesting featureof this model is that its nonlinear dynamics can beexactly solved as a consequence of a large num-ber of conservation laws that constrain the systemto follow particular trajectories. More precisely,the gradient flow preserves the difference of theGramian matrices of the input and output weights,and its convergence to equilibrium depends onboth the magnitude of that difference (which isfixed at initialization) and the spectrum of the data.In addition, and generalizing prior work, we proveour results without assuming small, balanced orspectral initialization for the weights. Moreover,we establish interesting mathematical connectionsbetween matrix factorization problems and differ-ential equations of the Riccati type.

Author Information

Salma Tarmoun (Johns Hopkins University)
Guilherme Franca (UC Berkeley)
Benjamin Haeffele (Johns Hopkins University)
Rene Vidal (Johns Hopkins University, USA)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors