Approximating f -Divergences with Rank Statistics

We introduce a rank-statistic approximation of $f$-divergences that avoids explicit density-ratio estimation by working directly with the distribution of ranks. For a resolution parameter $K$, we map the mismatch between two univariate distributions $\mu$ and $\nu$ to a rank histogram on $\{ 0, \ldots, K\}$ and measure its deviation from uniformity via a discrete $f$-divergence, yielding a rank-statistic divergence estimator. We prove that the resulting estimator of the divergence is monotone in $K$, is always a lower bound of the true $f$-divergence, and we establish quantitative convergence rates for $K\to\infty$ under mild regularity of the quantile-domain density ratio. To handle high-dimensional data, we define the sliced rank-statistic $f$-divergence by averaging the univariate construction over random projections, and we provide convergence results for the sliced limit as well. We also derive finite-sample deviation bounds along with asymptotic normality results for the estimator. Finally, we empirically validate the approach by benchmarking against neural baselines and illustrating its use as a learning objective in generative modeling experiments.

Many machine learning systems need to compare two datasets. For example, we may want to check whether images produced by a generative model resemble real images, or whether two samples come from the same source. Existing comparison methods can be hard to use because they often require estimating hidden probability information from data, which becomes unreliable for complex datasets such as images. We propose a simpler approach based on ranks. Instead of trying to estimate these hidden probabilities directly, we look at where samples from one dataset fall relative to samples from another. If the datasets are similar, these relative positions should be evenly spread; if they are different, clear patterns appear. Our paper turns this idea into a general method for comparing datasets. We prove that the method behaves reliably, extend it to high-dimensional data by looking at many one-dimensional views, and show in experiments that it can give stable comparisons and useful learning signals for generative modeling.