Empirical Gaussian Processes

Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. Re-evaluating this challenge from a hierarchical Bayesian and function-space view, we study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data-generating process. We formulate the problem of learning the GP prior from independent datasets as maximum likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.

In science and engineering we often must predict how a new system behaves from just a handful of measurements, even though records exist from many similar systems. A popular tool for this, called a Gaussian process, is valued because it reports not only a prediction but also how uncertain that prediction is. However, it must be told in advance what patterns to expect, and people usually guess this from a short list of standard templates — a poor guess yields inaccurate predictions and misleading confidence. We show how to learn those expectations directly from the historical records instead of guessing. The high-level recipe is simple: treat each past dataset as one example of a curve, then measure both the average curve and how such curves tend to vary together. We prove that this reconstructs the true underlying pattern as more examples accumulate, and we give an exact procedure that still works when past data is noisy or only partially recorded. On a new task with little data, our method delivers more accurate predictions and more honest uncertainty than hand-tuned alternatives and competing approaches. It also runs at a fraction of their computational cost, making reliable predictions easier whenever relevant history exists.