LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry

Data-adaptive two-sample testing assesses if two samples come from the same distribution, using a discrepancy learned from the data (e.g., via kernel-based feature representations). Such methods typically rely on data splitting to decouple learning from testing and control type I error. However, this paradigm is ill-suited to few-shot settings with severe sample-size imbalance: abundant reference samples are available, while only a handful of query samples arrive. In this paper, we show how this imbalance can be leveraged constructively. Using abundant reference data, we learn reference-dependent representations that summarize salient structure of the reference distribution and provide informative signals for detecting departures. We incorporate a collection of representation families that capture both global and local structure, and adaptively weight them using only reference samples via an uncertainty-guided principle. Theoretically, we establish permutation-based type I error control and show consistency of the aggregated test: as the sample sizes grow, the test power converges to one whenever the representation set contains at least one consistent representation. Empirically, our aggregation achieves strong performance across a range of benchmarks while retaining type I error control.

We study how to tell whether a small new batch of data looks like a large collection of trusted historical data. This problem appears in many real-world machine learning systems, such as detecting dataset shift, adversarial inputs, unusual user behavior, or other unexpected changes after deployment. Most existing adaptive testing methods need to learn from both the historical data and the new data. However, when only a few new examples are available, using them for learning can be unreliable and leaves too little data for testing. Our idea is instead to learn only from the abundant reference data. We build several reference-based signals that describe different aspects of the normal data distribution, such as global structure and local neighborhood patterns. We then combine these signals using an uncertainty-guided strategy and test whether the new data are still compatible with the reference data. Our method, LOTTERY, provides statistical error control and performs well when the new batch is very small.