Skip to yearly menu bar Skip to main content


On Dropout and Nuclear Norm Regularization

Poorya Mianjy · Raman Arora

Pacific Ballroom #79

Keywords: [ Deep Learning Theory ]

Abstract: We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss. We show that (a) the explicit regularizer is composed of an $\ell_2$-path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regularizer is the squared nuclear norm of the network map, and (c) for a sufficiently large dropout rate, we characterize the global optima of the dropout objective. We validate our theoretical findings with empirical results.

Live content is unavailable. Log in and register to view live content