Timezone: »
Gaussian process hyperparameter optimization requires linear solves with, and log-determinants of, large kernel matrices. Iterative numerical techniques are becoming popular to scale to larger datasets, relying on the conjugate gradient method (CG) for the linear solves and stochastic trace estimation for the log-determinant. This work introduces new algorithmic and theoretical insights for preconditioning these computations. While preconditioning is well understood in the context of CG, we demonstrate that it can also accelerate convergence and reduce variance of the estimates for the log-determinant and its derivative. We prove general probabilistic error bounds for the preconditioned computation of the log-determinant, log-marginal likelihood and its derivatives. Additionally, we derive specific rates for a range of kernel-preconditioner combinations, showing that up to exponential convergence can be achieved. Our theoretical results enable provably efficient optimization of kernel hyperparameters, which we validate empirically on large-scale benchmark problems. There our approach accelerates training by up to an order of magnitude.
Author Information
Jonathan Wenger (University of Tübingen)
Geoff Pleiss (Columbia University)
Philipp Hennig (University of Tuebingen)
John Cunningham (Columbia)
Jacob Gardner (University of Pennsylvania)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Oral: Preconditioning for Scalable Gaussian Process Hyperparameter Optimization »
Tue. Jul 19th 06:00 -- 06:20 PM Room Room 307
More from the Same Authors
-
2023 : Black Box Adversarial Prompting for Foundation Models »
Natalie Maus · Patrick Chao · Eric Wong · Jacob Gardner -
2023 : Practical and Asymptotically Exact Conditional Sampling in Diffusion Models »
Brian Trippe · Luhuan Wu · Christian Naesseth · David Blei · John Cunningham -
2023 Oral: Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference »
Kyurae Kim · Kaiwen Wu · Jisu Oh · Jacob Gardner -
2023 Poster: Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference »
Kyurae Kim · Kaiwen Wu · Jisu Oh · Jacob Gardner -
2022 Poster: Scaling Structured Inference with Randomization »
Yao Fu · John Cunningham · Mirella Lapata -
2022 Spotlight: Scaling Structured Inference with Randomization »
Yao Fu · John Cunningham · Mirella Lapata -
2022 Poster: Probabilistic ODE Solutions in Millions of Dimensions »
Nicholas Krämer · Nathanael Bosch · Jonathan Schmidt · Philipp Hennig -
2022 Poster: Fenrir: Physics-Enhanced Regression for Initial Value Problems »
Filip Tronarp · Nathanael Bosch · Philipp Hennig -
2022 Poster: Variational nearest neighbor Gaussian process »
Luhuan Wu · Geoff Pleiss · John Cunningham -
2022 Spotlight: Variational nearest neighbor Gaussian process »
Luhuan Wu · Geoff Pleiss · John Cunningham -
2022 Spotlight: Fenrir: Physics-Enhanced Regression for Initial Value Problems »
Filip Tronarp · Nathanael Bosch · Philipp Hennig -
2022 Spotlight: Probabilistic ODE Solutions in Millions of Dimensions »
Nicholas Krämer · Nathanael Bosch · Jonathan Schmidt · Philipp Hennig -
2021 Poster: Bias-Free Scalable Gaussian Processes via Randomized Truncations »
Andres Potapczynski · Luhuan Wu · Dan Biderman · Geoff Pleiss · John Cunningham -
2021 Spotlight: Bias-Free Scalable Gaussian Processes via Randomized Truncations »
Andres Potapczynski · Luhuan Wu · Dan Biderman · Geoff Pleiss · John Cunningham -
2020 Poster: The continuous categorical: a novel simplex-valued exponential family »
Elliott Gordon-Rodriguez · Gabriel Loaiza-Ganem · John Cunningham -
2020 Poster: Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks »
Agustinus Kristiadi · Matthias Hein · Philipp Hennig -
2020 Poster: Differentiable Likelihoods for Fast Inversion of 'Likelihood-Free' Dynamical Systems »
Hans Kersting · Nicholas Krämer · Martin Schiegg · Christian Daniel · Michael Schober · Philipp Hennig -
2019 Poster: Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography »
Andrew Miller · Ziad Obermeyer · John Cunningham · Sendhil Mullainathan -
2019 Oral: Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography »
Andrew Miller · Ziad Obermeyer · John Cunningham · Sendhil Mullainathan