Timezone: »

Bayesian Optimization Meets Bayesian Optimal Stopping
Zhongxiang Dai · Haibin Yu · Bryan Kian Hsiang Low · Patrick Jaillet

Wed Jun 12 03:10 PM -- 03:15 PM (PDT) @ Room 101

Bayesian optimization (BO) is a popular paradigm for optimizing the hyperparameters of machine learning (ML) models due to its sample efficiency. Many ML models require running an iterative training procedure (e.g., stochastic gradient descent). This motivates the question whether information available during the training process (e.g., validation accuracy after each epoch) can be exploited for improving the epoch efficiency of BO algorithms by early-stopping model training under hyperparameter settings that will end up under-performing and hence eliminating unnecessary training epochs. This paper proposes to unify BO (specifically, Gaussian process-upper confidence bound (GP-UCB)) with Bayesian optimal stopping (BO-BOS) to boost the epoch efficiency of BO. To achieve this, while GP-UCB is sample-efficient in the number of function evaluations, BOS complements it with epoch efficiency for each function evaluation by providing a principled optimal stopping mechanism for early stopping. BO-BOS preserves the (asymptotic) no-regret performance of GP-UCB using our specified choice of BOS parameters that is amenable to an elegant interpretation in terms of the exploration-exploitation trade-off. We empirically evaluate the performance of BO-BOS and demonstrate its generality in hyperparameter optimization of ML models and two other interesting applications.

Author Information

Zhongxiang Dai (National University of Singapore)
Haibin Yu (National University of Singapore)
Bryan Kian Hsiang Low (National University of Singapore)
Patrick Jaillet (MIT)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors