No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference

Prediction-Powered Inference (PPI) is a popular strategy for combining gold-standard and possibly noisy pseudo-labels to perform statistical estimation. Prior work has shown an asymptotic \textit{free lunch} for PPI++, an adaptive form of PPI, showing that the \textit{asymptotic} variance of PPI++ is always less than or equal to the variance obtained from using gold-standard labels alone. Notably, this result holds \textit{regardless of the quality of the pseudo-labels}. In this work, we demystify this result by conducting an exact finite-sample analysis of the estimation error of PPI++ on the mean estimation problem. We give a \textit{no free lunch} result, characterizing the settings (and sample sizes) where PPI++ has provably worse estimation error than using gold-standard labels alone. Specifically, PPI++ will outperform if and only if the correlation between pseudo- and gold-standard is above a certain level that depends on the number of labeled samples ($n$). In some cases our results simplify considerably: For Gaussian data, for instance, the correlation must be at least $1/\sqrt{n - 2}$ in order to see improvement. More broadly, by providing exact non-asymptotic expressions for the variance of PPI++ under sample splitting, we aim to empower practitioners to transparently reason about the benefits of PPI++ in specific applications. In experiments, we illustrate that our theoretical findings hold on real-world datasets.

Imagine you are trying to estimate the average of something important (prevalence of a disease, the fraction of times a product works, etc.). The standard approach is to hand-label a small set of examples and average them. However, hand-labeling is arduous and does not scale. A tempting shortcut has recently been to use LLMs to label a much larger set of examples and include those in the mix. Given the LLM labels could be wrong, a popular recent method, PPI++, provides an approach which weighs the LLM labels based on an estimate of their quality. This method makes a striking claim that it would never do worse than simply ignoring the LLM labels no matter how noisy they are. In our paper, we show that this is only true when you already have lots of hand-labeled data, which is exactly the situation where you would not need the LLMs help in the first place. When the hand-labeled data is actually scarce, the "never-do-worse" condition breaks, and clear conditions emerge where PPI++ actually helps vs hurts. We exactly characterize those conditions, showing a requirement on the correlation of the synthetic labels to be above a threshold which depends inversely on the number of hand-labeled examples available. The intuition is that if the quality of the pseudo-labels is already high, the \texit{estimate} of this quality need not be super precise. If the quality of the pseudo-labels is low, the estimate of this quality needs to reliably down-weight the contribution of the synthetic-labels. A simple rule of thumb exists for distributions routinely encountered in practice that PPI++ only works when the proxy-label's correlation is at least 1 divided by the square root of the sample-size. The good news is that this bar is not impossibly high, and informs a practical tool: given reasonable assumptions that the practitioner is willing to make on the quality of the proxy-labels, our framework is able to identify a sample-size where they can safely use synthetic labels.