Conformal C2ST: Turning weak classifiers into strong two-sample tests

The two-sample testing problem, a fundamental task in statistics and machine learning, seeks to determine whether two sets of samples, drawn from underlying distributions $p$ and $q$, are in fact identically distributed (i.e.~whether $p=q$). A popular and intuitive approach is the classifier two-sample test (C2ST), where a classifier is trained to distinguish between samples from $p$ and $q$. Yet despite simplicity of the C2ST, its reliability hinges on access to a near-Bayes-optimal classifier, a requirement that is rarely met and difficult to verify. This raises a major open question: can a weak classifier still be useful for two-sample testing? We show that the answer is a definitive yes. Building on the work of Hu & Lei (2024), we analyze two conformal variants of the C2ST that convert the scores from any trained classifier---even if weak, biased, or overfit---into exact, finite-sample p-values. We establish two key theoretical properties of the conformal C2ST: (i) finite-sample Type-I error control, and (ii) non-trivial power that degrades gently in tandem with the error of the trained classifier. The upshot is that even poorly performing classifiers can yield powerful and reliable two-sample tests. This general framework finds a powerful application in Bayesian inference, particularly for validating Neural Posterior Estimation (NPE) models, where the task of comparing a learned posterior approximation $q(\theta \mid y)$ to the true posterior $p(\theta \mid y)$ can be framed as a two-sample test. Empirically, the Conformal C2ST outperforms classical discriminative tests across a wide range of benchmarks for this task. Our results establish the conformal C2ST as a practical, theoretically grounded diagnostic tool.

Scientists increasingly use AI models that imitate complex real-world processes — for instance, generating videos, text, or even reconstructing a galaxy's true appearance from light that was distorted on its way to our telescopes. But how do we know such an answer is genuinely right, rather than subtly wrong in a way that could mislead a scientific conclusion? A common check trains a second program to separate the model's outputs from real examples: if it cannot tell them apart, the model passes. The trouble is that this only works when that program is very accurate, and in practice it is often weak. A weak program can wrongly clear a flawed model, so a "pass" may simply mean the check was too poor to notice the problem — not that the model is sound. We show that even a weak program can be turned into a reliable, sensitive test. Instead of relying on its yes-or-no judgments, we use only the rough ordering it assigns to examples — something even a poor program gets partly right — and a simple post-processing converts these rankings into trustworthy evidence with guaranteed error rates. Our test catches subtle mistakes that other methods miss and stays dependable even when the program behind it is unreliable, giving scientists a practical safety check for the AI models they rely on.