Understanding the difficulties of posterior predictive estimation

Predictive posterior densities (PPDs) are essential in approximate inference for quantifying predictive uncertainty and comparing inference methods. Typically, PPDs are estimated by simple Monte Carlo (MC) averages. In this paper, we expose a critical under-recognized issue: the signal-to-noise ratio (SNR) of the simple MC estimator can sometimes be extremely low, leading to unreliable estimates. Our main contribution is a theoretical analysis demonstrating that even with exact inference, SNR can decay rapidly with an increase in (a) the mismatch between training and test data, (b) the dimensionality of the latent space, or (c) the size of test data relative to training data. Through several examples, we empirically verify these claims and show that these factors indeed lead to poor SNR and unreliable PPD estimates (sometimes, estimates are off by hundreds of nats even with a million samples). While not the primary focus, we also explore an adaptive importance sampling approach as an illustrative way to mitigate the problem, where we learn the proposal distribution by maximizing a variational proxy to the SNR. Taken together, our findings highlight an important challenge and provide essential insights for reliable estimation.

Posterior predictive distribution (PPD) is an important quantity in Bayesian inference as it is used for making predictions and comparing performance of inference methods. In this work, we uncover that naive methods of estimating PPD can be unreliable. Our primary contribution is a theoretical analysis to understand what factors lead to poor estimation when using the naive method. Based on this analysis, we propose a new technique to estimate PPD and then demonstrate how this proposed method can vastly improve the quality of estimates while using fewer resources (time and compute).