Are Two Datasets Close Enough With Statistical Significance? A Kernel Distributional Closeness Testing Approach

Are two distributions close to each other with statistical significance? *Distribution closeness testing* (DCT) formalizes this question by testing whether the distance between a distribution pair is at least $\epsilon$-far. Existing DCT methods mainly measure discrepancies between a distribution pair defined on discrete spaces (e.g., using total variation), which limits their applications to complex data (e.g., images). To extend DCT to more types of data, a natural idea is to introduce *maximum mean discrepancy* (MMD), a powerful measurement of the distributional discrepancy between two complex distributions, into DCT scenarios. However, the empirical results indicate that many distribution pairs can have the same MMD value despite having different norms in the same *reproducing kernel Hilbert space* (RKHS), and these pairs may exhibit different finite-sample distinguishability and reflect different practical closeness levels, making MMD less informative in DCT. To mitigate the issue, we design a new measurement of distributional discrepancy, *norm-adaptive MMD* (NAMMD), which scales MMD's value using the RKHS norms of distributions. Based on the asymptotic distribution of NAMMD, we finally propose the NAMMD-based DCT to assess the closeness level of a distribution pair. Theoretically, we prove that NAMMD-based DCT has higher test power compared to MMD-based DCT, with bounded type-I error, which is also validated by extensive experiments on many types of data (e.g., synthetic noise, real images). Our code is available at: https://github.com/zhijianzhouml/NAMMD.

Machine learning models are often used on data that are not exactly the same as the data they were trained on. In many cases, small differences between datasets may be acceptable, while large differences may require extra checking, retraining, or model adaptation. This paper studies how to decide, with statistical confidence, whether two datasets are close enough to each other. Existing methods for this problem mainly work for simple discrete data and are less suitable for complex data such as images. We propose a kernel-based method that can compare more complex datasets by measuring not only how different they are, but also how informative that difference is for deciding closeness. Our method provides statistical error control and performs well in experiments on both synthetic and real image datasets. It can help practitioners monitor distribution shifts, compare datasets, and decide when a model may need further validation or adaptation.