Singular Bayesian Neural Networks

Bayesian neural networks promise calibrated uncertainty but require $O(mn)$ parameters for standard mean-field Gaussian posteriors. We argue this cost is often unnecessary, particularly when weight matrices exhibit fast singular value decay. By parameterizing weights as $W = AB^{\top}$ with $A \in \mathbb{R}^{m \times r}$, $B \in \mathbb{R}^{n \times r}$, we induce a posterior that is \emph{singular} with respect to the Lebesgue measure, concentrating on the rank-$r$ manifold. This singularity captures structured weight correlations through shared latent factors, geometrically distinct from mean-field's independence assumption. We derive PAC-Bayes generalization bounds whose complexity term scales as $\sqrt{r(m+n)}$ instead of $\sqrt{m n}$, and prove loss bounds that decompose the error into optimization and rank-induced bias using the Eckart-Young-Mirsky theorem. We further adapt recent Gaussian complexity bounds for low-rank deterministic networks to Bayesian predictive means. Empirically, across MLPs, LSTMs, and Transformers on standard benchmarks, our method achieves competitive predictive performance while using up to $33\times$ fewer parameters than 5-member Deep Ensembles. It substantially improves OOD detection and often improves calibration relative to mean-field and perturbation baselines, while Deep Ensembles can still be stronger on in-distribution likelihood-based metrics.

Artificial intelligence systems now play a central role in daily life, from medical diagnosis and content moderation to autonomous driving and AI agents that act on our behalf. As these systems take on more responsibility, they must become more trustworthy. In particular, they should recognize when they are uncertain, rather than giving confident answers in situations where they may be wrong. Recent examples of large models giving incorrect factual answers with high stated confidence illustrate the problem: neural networks can be confidently wrong, and in high-stakes settings that failure can be costly. Bayesian deep learning offers one way to address this problem. A standard neural network stores a single fixed value for each weight, where weights are the numbers that determine how the model transforms inputs into predictions. In Bayesian deep learning, each weight is represented by a probability distribution instead. During training, the model learns not only a typical value for each weight, but also how uncertain that value is. At prediction time, this allows the model to express uncertainty, especially on inputs that differ from those seen during training. The challenge is that this extra uncertainty comes at a high computational cost. In a common approach called mean-field variational inference, each weight is described by both a mean and a variance. This roughly doubles the number of learned parameters compared with an ordinary network, and also increases the cost of training and prediction. For small models this is manageable, but modern neural networks often contain hundreds of millions or billions of weights. At that scale, standard Bayesian methods can become too expensive to use in practice. This paper introduces a low-rank factorization approach to overcome that barrier. Each layer's weights form a large matrix with m rows and n columns. Instead of storing this full matrix, we write it as the product of two smaller matrices that share a common inner dimension r, called the rank. This reduces the number of parameters per layer from m × n to r × (m + n). The rank r is a value we choose, and the smaller we can set it while preserving accuracy, the greater the saving. In practice, it can be set far below m and n, because most of the meaningful structure in these weight matrices lives in only a few dominant directions, so the compression loses little while saving a great deal. This reduces not only the parameter count but also the computation time, and as a byproduct it lets the weights remain correlated rather than treating each one as independent, which results in a more coherent propagation of uncertainty across the network. We prove that this is more than a convenient shortcut: the low-rank version carries strong generalization guarantees, with the compression error controlled by how quickly the matrix's minor directions fade out. We then test it on intensive care mortality prediction, air quality forecasting, and movie review sentiment classification. The low-rank model performs better than far larger and costlier methods in key settings, including deep ensembles, while using only a small fraction of the parameters. In one case, it uses about thirty times fewer parameters than an ensemble of five networks. It is particularly strong at flagging inputs that differ from its training data and at producing well-calibrated uncertainty. Taken together, these results show that low-rank factorization makes it possible to scale Bayesian deep learning to modern architectures, yielding uncertainty-aware AI systems that are more robust to distributional shift and more trustworthy at deployment. The result is uncertainty-aware deep learning that is smaller, theoretically grounded, empirically strong, and practical for the high-stakes settings that need it most.