Timezone: »

 
Spotlight
Learning Neural Network Subspaces
Mitchell Wortsman · Maxwell Horton · Carlos Guestrin · Ali Farhadi · Mohammad Rastegari

Tue Jul 20 05:45 PM -- 05:50 PM (PDT) @
in Deep Learning 2 »

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.

Keywords: Deep Learning · Representation Learning · Algorithms · Dialog- or Communication-Based Learning

Author Information

Mitchell Wortsman (University of Washington)
Maxwell Horton (Apple)
Carlos Guestrin (Stanford University & Apple)
Ali Farhadi (University of Washington, Allen Institue for AI)
Mohammad Rastegari (University of Washington)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors