Semi-Supervised Hypothesis Testing by Betting on Predictions

We introduce a testing-by-betting framework that leverages predictions on unlabeled data to enhance the power of sequential hypothesis testing. Given limited samples from the joint distribution of $(X,Y)$, and additional unlabeled samples from the marginal of $X$, we ask how unlabeled data can be used to hypothesize about the distribution of $Y$, and the conditional distribution of $Y\mid X$. We introduce an e-statistic and use it to construct a sequential test. Under standard distributional assumptions---label shift or concept shift---we establish that the test is anytime valid. Furthermore, we show that for binary data, the e-statistic has non-trivial power. Crucially, our approach retains these properties even when the underlying predictions are inaccurate. Through simulations and applications to large language models evaluation, we demonstrate power gains over baseline approaches, including prediction-powered inference. These gains persist even with relatively limited unlabeled data and when predictions have low accuracy due to weak correlation between $X$ and $Y$.

Statistical hypothesis testing is a fundamental tool for determining whether observed data provide sufficient evidence to support a scientific claim. In many modern applications, obtaining labels from humans is expensive and time-consuming, while large amounts of unlabeled data are readily available. This raises a natural question: can machine learning models be used to predict labels for unlabeled data and still support statistically reliable conclusions, even when those predictions are imperfect? In this work, we develop a new statistical testing method that uses predictions from unlabeled data to strengthen the evidence gathered during the test. Importantly, the method remains statistically reliable even when predictions are biased and imperfect. We show that our method improves the power of statistical tests in real-world applications such as large language model evaluation. In particular, it can detect meaningful improvements using fewer expensive human annotations, potentially reducing evaluation costs for modern AI systems and other data-driven applications.