Predictive variational inference: Learn the predictively optimal posterior distribution

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

To let a machine learning model admit epistemic humility when it is incomplete, we provide a family of methods that find a distribution of models that, when combined, gives the best prediction. The distribution can be used to detect the incompleteness of a single model. A broad distribution indicates that the model is incomplete for the data. A narrow distribution indicates that the model is complete for the data. The procedure also augments the predictive ability of incomplete models. When optimizing for prediction, a distribution of models is more "complete" than a single model. Furthermore, when we care about different prediction targets, such as the mean or the median, we may use different objectives in the family. We demonstrate that our method can be used to detect incompleteness in a golf putting model and an election model. We also show that it can work with biological simulators to solve for molecular parameters. We have additional benchmarks showing that we achieve the desired prediction goals.