Taking the GP Out of the Loop

Bayesian optimization (BO) has traditionally solved black-box problems where function evaluation is expensive and, therefore, observations are few. Recently, however, there has been growing interest in applying BO to problems where function evaluation is cheaper and observations are more plentiful. In this regime, scaling to many observations $N$ is impeded by Gaussian-process (GP) surrogates: GP hyperparameter fitting scales as $\mathcal{O}(N^3)$ (reduced to roughly $\mathcal{O}(N^2)$ in modern implementations), and it is repeated at every BO iteration. Many methods improve scaling at acquisition time, but hyperparameter fitting still scales poorly, making it the bottleneck. We propose Epistemic Nearest Neighbors (ENN), a lightweight alternative to GPs that estimates function values and uncertainty (epistemic and aleatoric) from $K$-nearest-neighbor observations. ENN scales as $\mathcal{O}(N)$ for both fitting and acquisition. Our BO method, TuRBO-ENN, replaces the GP surrogate in TuRBO with ENN and its Thompson-sampling acquisition with $\mathrm{UCB} = \mu(x) + \sigma(x)$. For the special case of noise-free problems, we can omit fitting altogether by replacing $\mathrm{UCB}$ with a non-dominated sort over $\mu(x)$ and $\sigma(x)$. We show empirically that TuRBO-ENN reduces proposal time (i.e., fitting time + acquisition time) by one to two orders of magnitude compared to TuRBO at up to 50,000 observations without sacrificing solution quality.

Bayesian optimization is automated trial-and-error for tuning simulators or robots. At each iteration of the search, it chooses which new settings to test using the measurements gathered so far. Older tools assumed each measurement was expensive, but today one run can yield tens of thousands of cheap measurements. Popular software packages still rebuild, on every iteration, a heavy model (often a Gaussian process) over all past measurements, and that planning cost rises sharply as more measurements accumulate. Epistemic Nearest Neighbors (ENN) forecasts new measurements and their uncertainty using only a handful of past measurements taken at settings most similar to those the optimizer is about to try, so work grows linearly with the number of measurements instead of quadratically. We combine ENN with a search (as in TuRBO) that keeps zooming in on the best neighborhoods, forming TuRBO-ENN. Noisy simulations use two simple calibration knobs; noise-free simulations can skip calibration. On five realistic example problems, TuRBO-ENN matched a strong Gaussian-process optimizer on solution quality while cutting the time spent planning each iteration by about ten- to a hundred-fold at large scale, speeding large tuning jobs and trimming wasted compute. Open code: ennbo at https://github.com/yubo-research/enn