Timezone: »

 
Spotlight
Bayesian Deep Learning via Subnetwork Inference
Erik Daxberger · Eric Nalisnick · James Allingham · Javier Antorán · Jose Miguel Hernandez-Lobato

Tue Jul 20 06:40 AM -- 06:45 AM (PDT) @
in Deep Learning (Bayesian) »

The Bayesian paradigm has the potential to solve core issues of deep neural networks such as poor calibration and data inefficiency. Alas, scaling Bayesian inference to large weight spaces often requires restrictive approximations. In this work, we show that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors. The other weights are kept as point estimates. This subnetwork inference framework enables us to use expressive, otherwise intractable, posterior approximations over such subsets. In particular, we implement subnetwork linearized Laplace as a simple, scalable Bayesian deep learning method: We first obtain a MAP estimate of all weights and then infer a full-covariance Gaussian posterior over a subnetwork using the linearized Laplace approximation. We propose a subnetwork selection strategy that aims to maximally preserve the model’s predictive uncertainty. Empirically, our approach compares favorably to ensembles and less expressive posterior approximations over full networks.

Keywords: Bayesian Deep Learning · Reinforcement Learning and Planning · Multi-Agent RL

Author Information

Erik Daxberger (University of Cambridge & MPI for Intelligent Systems, Tübingen)
Eric Nalisnick (University of Amsterdam)
James Allingham (University of Cambridge)
Javier Antorán (University of Cambridge)

I am a PhD student in Machine Learning at the University of Cambridge under the supervision of Dr. José Miguel Hernández-Lobato. I’m interested in Bayesian deep learning, representation learning, uncertainty in machine learning and information theory. I graduated from the University of Zaragoza in 2018 with an honorary distinction (“premio extraordinario”) in Telecommunications Engineering (EE/CS). I was awarded an MPhil in Machine Learning with distinction by the University of Cambridge in 2019. I also do freelance engineering consulting and am a co-founder of arisetech.es.

Jose Miguel Hernandez-Lobato (University of Cambridge)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors