Timezone: »

 
Oral
Dropout as a Structured Shrinkage Prior
Eric Nalisnick · Jose Miguel Hernandez-Lobato · Padhraic Smyth

Thu Jun 13 04:30 PM -- 04:35 PM (PDT) @ Grand Ballroom
in Bayesian Deep Learning »

Dropout regularization of deep neural networks has been a mysterious yet effective tool to prevent overfitting. Explanations for its success range from the prevention of "co-adapted" weights to it being a form of cheap Bayesian inference. We propose a novel framework for understanding multiplicative noise in neural networks, considering continuous distributions as well as Bernoulli noise (i.e. dropout). We show that multiplicative noise induces structured shrinkage priors on a network's weights. We derive the equivalence through reparametrization properties of scale mixtures and without invoking any approximations. Given the equivalence, we then show that dropout's Monte Carlo training objective approximates marginal MAP estimation. We extend this framework to ResNets, terming the prior "automatic depth determination" as it is the natural analog of "automatic relevance determination" for network depth. Lastly, we investigate two inference strategies that improve upon the aforementioned MAP approximation in regression benchmarks.

[ Slides] 

Author Information

Eric Nalisnick (University of Cambridge & DeepMind)
Jose Miguel Hernandez-Lobato (University of Cambridge)
Padhraic Smyth (UC Irvine)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors