Self-Supervised Variational Priors for Robust Bayesian Inference

The standard in Bayesian Deep Learning is to use zero-mean isotropic Gaussian priors over neural network weights. When the likelihood is reliable (\eg, with abundant clean labels), these priors matter little. However, when the likelihood is unreliable due to imperfect data, the weakness of standard priors becomes apparent, resulting in poor robustness and uncertainty estimates. To address this, we propose to replace the standard priors with self-supervised variational posteriors. Specifically, we propose a variational method based on instance discrimination. Our objective is derived from the evidence lower bound and made scalable through approximations. Empirically, our priors enhance data-efficiency, robustness to label noise, generalization and calibration on the CIFAR datasets.