Timezone: »
Oral
On Dropout and Nuclear Norm Regularization
Poorya Mianjy · Raman Arora
We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with the squared loss. We show that (a) the explicit regularizer is composed of an $\ell_2$-path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regularizer is the squared nuclear norm of the network map, and (c) for a sufficiently large dropout rate, we characterize the global optima of the dropout objective. We validate our theoretical findings with empirical results.
Author Information
Poorya Mianjy (Johns Hopkins University)
Raman Arora (Johns Hopkins University)
Raman Arora received his M.S. and Ph.D. degrees in Electrical and Computer Engineering from the University of Wisconsin-Madison in 2005 and 2009, respectively. From 2009-2011, he was a Postdoctoral Research Associate at the University of Washington in Seattle and a Visiting Researcher at Microsoft Research Redmond. Since 2011, he has been with Toyota Technological Institute at Chicago (TTIC). His research interests include machine learning, speech recognition and statistical signal processing.
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: On Dropout and Nuclear Norm Regularization »
Wed. Jun 12th 01:30 -- 04:00 AM Room Pacific Ballroom #79
More from the Same Authors
-
2021 Poster: Robust Learning for Data Poisoning Attacks »
Yunjuan Wang · Poorya Mianjy · Raman Arora -
2021 Spotlight: Robust Learning for Data Poisoning Attacks »
Yunjuan Wang · Poorya Mianjy · Raman Arora -
2021 Poster: Dropout: Explicit Forms and Capacity Control »
Raman Arora · Peter Bartlett · Poorya Mianjy · Nati Srebro -
2021 Spotlight: Dropout: Explicit Forms and Capacity Control »
Raman Arora · Peter Bartlett · Poorya Mianjy · Nati Srebro -
2020 Poster: FetchSGD: Communication-Efficient Federated Learning with Sketching »
Daniel Rothchild · Ashwinee Panda · Enayat Ullah · Nikita Ivkin · Ion Stoica · Vladimir Braverman · Joseph E Gonzalez · Raman Arora -
2018 Poster: On the Implicit Bias of Dropout »
Poorya Mianjy · Raman Arora · Rene Vidal -
2018 Oral: On the Implicit Bias of Dropout »
Poorya Mianjy · Raman Arora · Rene Vidal -
2018 Poster: Streaming Principal Component Analysis in Noisy Setting »
Teodor Vanislavov Marinov · Poorya Mianjy · Raman Arora -
2018 Poster: Stochastic PCA with $\ell_2$ and $\ell_1$ Regularization »
Poorya Mianjy · Raman Arora -
2018 Oral: Streaming Principal Component Analysis in Noisy Setting »
Teodor Vanislavov Marinov · Poorya Mianjy · Raman Arora -
2018 Oral: Stochastic PCA with $\ell_2$ and $\ell_1$ Regularization »
Poorya Mianjy · Raman Arora