Policy Search via Bayesian Optimization with Temporal Difference Gaussian Processes

Bayesian optimization (BO) is a method commonly used for policy search in problems with low-dimensional policy parameterizations. While it is generally considered data-efficient, existing BO approaches are agnostic to the sequential structure of the optimization objective induced by policy roll-outs. Thereby, valuable information is discarded that could improve the convergence of BO. We address this inefficiency by developing and rigorously analyzing a novel approach for BO that relies on a temporal difference learning formulation for discounted infinite-horizon value functions based on Gaussian process (GP) regression. We derive learning error bounds for the proposed temporal difference GPs, such that we can exploit upper confidence bounds to analyze the cumulative regret of our BO approach. This analysis is further refined by bounding the maximal information gain for our temporal difference GP model. In a comparison with relevant baseline methods, we demonstrate the practical advantages of our method.

Teaching machines, like robots, to perform tasks often involves trial and error. A popular strategy for this is Bayesian Optimization, which helps machines to guess the best settings based on past attempts. However, standard versions of this strategy have a blind spot: they ignore the step-by-step, sequential nature of a machine's actions, wasting valuable information. Instead of just looking at the final result of an attempt, our method exploits the step-by-step changes, i.e., it accumulates how one specific action impacts the very next moment and thereby the whole outcome. By combining this approach with a system that measures its own uncertainty, machines can learn the best settings much faster and with fewer attempts. The paper proves mathematically that our method is highly efficient. Moreover, it provides results for tests in computer simulations, such as learning to balance a pendulum, where it consistently outperforms existing techniques. Ultimately, this research offers a faster, more reliable way to train automated systems.