Simulation-based inference is quickly becoming a standard technique to analyse data from cosmological surveys. While there has been significant recent advances in core density estimation models, applications of such techniques to real data are entirely reliant on the generalization power of neural networks, which is largely unconstrained outside the training distribution. Due to our inability to generate simulations which perfectly approximate real observations, and the large computational expense of simulating all the parameter space, simulation-based inference methods in Cosmology are vulnerable to such generalization issues.Here, we discuss the effects of both of these issues, and show how using Bayesian neural networks can mitigate biases and lead to more reliable inference outside the training set. We introduce cosmoSWAG, the first application of Stochastic Weight Averaging to cosmology, and apply it to simulation-based inference from the cosmic microwave background.