Scalable and Stable Estimation of Amari $\alpha$-Divergence using Random Fourier Features

Reliable estimation of Amari $\alpha$-divergences underpins variational inference, yet unconstrained neural critics are notoriously prone to instability. We propose a scalable estimator by constraining the critic to a Reproducing Kernel Hilbert Space (RKHS) ball and approximating the kernel via band-limited Random Fourier Features (RFF). This formulation yields a linear-time objective amenable to mini-batch stochastic optimization while avoiding the cubic complexity of Gram-matrix methods. We present a unified analysis based on a four-term error decomposition—comprising RKHS approximation, feature discretization, statistical deviation, and optimization residual. Under a spectral source condition, we derive non-asymptotic bounds establishing that the RKHS approximation bias scales as $\mathcal{O}(R^{-\gamma})$, the RFF discretization error as $\mathcal{O}(R D^{-1/2})$, and the statistical error as $\mathcal{O}(R n^{-1/2})$. We further show that statistical non-degeneracy induces intrinsic local curvature, enabling our proposed Armijo-SGD to achieve local linear convergence. Empirical evaluations demonstrate that the RFF-RKHS estimator outperforms varying-representation baselines in stability, and applying this spectral regularization to GAN critics significantly enhances the capture of high-frequency data components.

Many machine-learning systems need a reliable way to compare two sets of data, but current methods can become unstable or too expensive when datasets are large. This paper introduces a more stable and scalable way to make that comparison by combining strong mathematical safeguards with a lightweight approximation that is fast enough for modern large-scale training. In practical terms, the method keeps the model from overreacting during training while also avoiding the heavy computational cost of older kernel-based approaches. The authors show, both in theory and in experiments, that their method gives more accurate and more stable estimates than common neural-network baselines, especially as the problem becomes larger or higher-dimensional. They also apply the same idea to image generation and find that it helps the model capture finer visual details and improves training stability. Overall, the work offers a way to make an important building block of machine learning more dependable and more practical for real-world use.