Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling

Gaussian Process (GP) Variational Autoencoders (VAEs) extend standard VAEs by replacing the fully factorised Gaussian prior with a GP prior, thereby capturing richer correlations among latent variables. However, performing exact GP inference in large-scale GPVAEs is computationally prohibitive, often forcing existing approaches to rely on restrictive kernel assumptions or large sets of inducing points. In this work, we propose a neighbour-driven approximation strategy that exploits local adjacencies in the latent space to achieve scalable GPVAE inference. By confining computations to the nearest neighbours of each data point, our method preserves essential latent dependencies, allowing more flexible kernel choices and mitigating the need for numerous inducing points. Through extensive experiments on tasks including representation learning, data imputation, and conditional generation, we demonstrate that our approach outperforms other GPVAE variants in both predictive performance and computational efficiency.

Machines often learn from data that comes with a natural order—for example, the frames of a video, sensor readings over time, or measurements across a map. To understand such data, researchers use a tool called a variational auto-encoder (VAE), which compresses each data point into a small “code” inside the model. Classic VAEs treat every point as independent, so they miss the fact that neighbouring points—adjacent frames, nearby sensors—are usually related.Our work shows how to keep those neighbour relationships without slowing the model down. We give each hidden code a short “friends list,” asking it to pay attention only to its closest neighbours. This tiny change lets the model keep the benefits of neighbourhood awareness while staying computationally efficient. We build two versions of the idea and test them on tasks such as repairing missing video frames, predicting future steps in simulated robots, and filling gaps in large environmental maps. In every case, the neighbour-aware models are faster and more accurate than earlier approaches.