Diffusion Gaussian Processes

We propose Diffusion Gaussian Processes (DGPs), a novel framework that integrates Gaussian processes (GPs) into Denoising Diffusion Probabilistic Models (DDPMs), leveraging DDPMs' generative flexibility while embedding GPs as denoisers to improve uncertainty quantification in regression with arbitrary likelihoods. While diffusion models demonstrate strong performance in modelling complex distributions, their uncertainty estimates are limited by point estimations of neural networks, whereas GPs provide principled uncertainty estimates but are limited by scalability and assumptions of homoscedastic Gaussian likelihoods. To enable practical and scalable implementations, we introduce two instantiations based on modern GP approximations: DDPM with sparse variational GP (D-SVGP) using inducing-point variational inference, and DDPM with GP using iterative linear system solvers like stochastic dual descent (D-SDD). We further propose a trajectory-based method to decompose predictive variance into epistemic and aleatoric components within the DGP framework. Experiments on synthetic datasets and UCI regression benchmarks show DGPs achieving comparable predictive performances to existing baselines.