Timezone: »

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling
Jiri Hron · Roman Novak · Jeffrey Pennington · Jascha Sohl-Dickstein

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #810

We introduce repriorisation, a data-dependent reparameterisation which transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. The repriorisation map acts directly on parameters, and its analytic simplicity complements the known neural network Gaussian process (NNGP) behaviour of wide BNNs in function space. Exploiting the repriorisation, we develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. This contrasts with the typically poor performance of MCMC in high dimensions. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks. Improvements are achieved at all widths, with the margin between reparametrised and standard BNNs growing with layer width.

Author Information

Jiri Hron (University of Cambridge)
Roman Novak (Google Brain)
Jeffrey Pennington (Google Brain)
Jascha Sohl-Dickstein (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors