Timezone: »

 
Spotlight
Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling
Jiri Hron · Roman Novak · Jeffrey Pennington · Jascha Sohl-Dickstein

Wed Jul 20 11:30 AM -- 11:35 AM (PDT) @ Room 301 - 303
in PM: Bayesian Models and Methods »

We introduce repriorisation, a data-dependent reparameterisation which transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. The repriorisation map acts directly on parameters, and its analytic simplicity complements the known neural network Gaussian process (NNGP) behaviour of wide BNNs in function space. Exploiting the repriorisation, we develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. This contrasts with the typically poor performance of MCMC in high dimensions. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks. Improvements are achieved at all widths, with the margin between reparametrised and standard BNNs growing with layer width.

Keywords: DL: Everything Else · PM: Bayesian Models and Methods · PM: Monte Carlo and Sampling Methods

Author Information

Jiri Hron (University of Cambridge)
Roman Novak (Google Brain)
Jeffrey Pennington (Google Brain)
Jascha Sohl-Dickstein (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors