Timezone: »

 
Poster
Linear Mode Connectivity and the Lottery Ticket Hypothesis
Jonathan Frankle · Gintare Karolina Dziugaite · Daniel Roy · Michael Carbin

Tue Jul 14 01:00 PM -- 01:45 PM & Wed Jul 15 02:00 AM -- 02:45 AM (PDT) @
in Poster Session 7 »

We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).

Keywords: Architectures · Optimization · Other · Deep Learning - General

Author Information

Jonathan Frankle (MIT CSAIL)
Gintare Karolina Dziugaite (Element AI)
Daniel Roy (University of Toronto; Vector Institute)
Michael Carbin (MIT)

More from the Same Authors