Timezone: »

 
Poster
How Good is the Bayes Posterior in Deep Neural Networks Really?
Florian Wenzel · Kevin Roth · Bastiaan Veeling · Jakub Swiatkowski · Linh Tran · Stephan Mandt · Jasper Snoek · Tim Salimans · Rodolphe Jenatton · Sebastian Nowozin

Tue Jul 14 09:00 AM -- 09:45 AM & Tue Jul 14 10:00 PM -- 10:45 PM (PDT) @
in Poster Session 3 »

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions when compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a ``cold posterior'' that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of cold posteriors.

Keywords: Approximate Inference · Bayesian Deep Learning · Bayesian Methods · Monte Carlo Methods · Deep Learning - General

Author Information

Florian Wenzel (Google Research)
Kevin Roth (ETH Zurich)
Bastiaan Veeling (University of Amsterdam)
Jakub Swiatkowski (University of Warsaw)
Linh Tran (Imperial College London)
Stephan Mandt (University of California, Irivine)

Stephan Mandt is an Assistant Professor of Computer Science at the University of California, Irvine. From 2016 until 2018, he was a Senior Researcher and head of the statistical machine learning group at Disney Research, first in Pittsburgh and later in Los Angeles. He held previous postdoctoral positions at Columbia University and at Princeton University. Stephan holds a PhD in Theoretical Physics from the University of Cologne. He is a Fellow of the German National Merit Foundation, a Kavli Fellow of the U.S. National Academy of Sciences, and was a visiting researcher at Google Brain. Stephan serves regularly as an Area Chair for NeurIPS, ICML, AAAI, and ICLR, and is a member of the Editorial Board of JMLR. His research is currently supported by NSF, DARPA, IBM, and Qualcomm.

Jasper Snoek (Google Brain)
Tim Salimans (Google)
Rodolphe Jenatton (Google Research)
Sebastian Nowozin (Microsoft Research)

More from the Same Authors