Timezone: »

On the Implicit Bias of Dropout
Poorya Mianjy · Raman Arora · Rene Vidal

Fri Jul 13 08:30 AM -- 08:40 AM (PDT) @ K11

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For shallow linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. We completely characterize the optimization landscape of single hidden-layer linear networks with dropout.

Author Information

Poorya Mianjy (Johns Hopkins University)
Raman Arora (Johns Hopkins University)
Raman Arora

Raman Arora received his M.S. and Ph.D. degrees in Electrical and Computer Engineering from the University of Wisconsin-Madison in 2005 and 2009, respectively. From 2009-2011, he was a Postdoctoral Research Associate at the University of Washington in Seattle and a Visiting Researcher at Microsoft Research Redmond. Since 2011, he has been with Toyota Technological Institute at Chicago (TTIC). His research interests include machine learning, speech recognition and statistical signal processing.

Rene Vidal (Johns Hopkins University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors