FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference

Prediction-powered inference (PPI) enables valid statistical inference by combining experimental data with machine learning predictions. When a sufficient number of high-quality predictions is available, PPI results in more accurate estimates and tighter confidence intervals than traditional methods. In this paper, we propose to inform the PPI framework with prior knowledge on the quality of the predictions. The resulting method, which we call frequentist, assisted by Bayes, PPI (FAB-PPI), improves over PPI when the observed prediction quality is likely under the prior, while maintaining its frequentist guarantees. Furthermore, when using heavy-tailed priors, FAB-PPI adaptively reverts to standard PPI in low prior probability regions. We demonstrate the benefits of FAB-PPI in real and synthetic examples.

Making accurate decisions requires a large amount of high-quality data, which is not always available. While machine learning (ML) predictions can help cheaply fill the gap, they may be biased, leading to incorrect conclusions. Prediction-powered inference (PPI) is a recent method that enables using ML predictions for making principled decisions by correcting for such bias using just a few gold standard observations. However, standard PPI does not take advantage of existing knowledge or "prior beliefs" about the likely quality of these ML predictions, which are often expected to be usually very good, but occasionally very off. Our work introduces FAB-PPI, a rigorous approach to incorporate such prior beliefs into standard PPI, while maintaining the latter's guarantees. FAB-PPI significantly improves over standard PPI when the predictions are good, and it reverts to standard PPI when they are very poor. Overall, FAB-PPI offers a principled way to leverage our knowledge about ML model performance for making decisions, with a built-in safety net.