Timezone: »

 
Oral
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
Shahar Azulay · Edward Moroshko · Mor Shpigel Nacson · Blake Woodworth · Nati Srebro · Amir Globerson · Daniel Soudry

Wed Jul 21 07:00 AM -- 07:20 AM (PDT) @
in Deep Learning Theory 3 »

Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to. In particular, it was shown that large initialization leads to the neural tangent kernel regime solution, whereas small initialization leads to so called ``rich regimes''. However, the initialization structure is richer than the overall scale alone and involves relative magnitudes of different weights and layers in the network. Here we show that these relative scales, which we refer to as initialization shape, play an important role in determining the learned model. We develop a novel technique for deriving the inductive bias of gradient-flow and use it to obtain closed-form implicit regularizers for multiple cases of interest.

Keywords: Deep Learning Theory · MCMC

Author Information

Shahar Azulay (TAU)
Edward Moroshko (Technion)
Mor Shpigel Nacson (Technion)
Blake Woodworth (Toyota Technological Institute at Chicago)
Nati Srebro (Toyota Technological Institute at Chicago)
Amir Globerson (Tel Aviv University, Google)
Daniel Soudry (Technion)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors