Timezone: »

Bayesian Optimization Meets Bayesian Optimal Stopping
Zhongxiang Dai · Haibin Yu · Bryan Kian Hsiang Low · Patrick Jaillet

Wed Jun 12 03:10 PM -- 03:15 PM (PDT) @ Room 101

Bayesian optimization (BO) is a popular paradigm for optimizing the hyperparameters of machine learning (ML) models due to its sample efficiency. Many ML models require running an iterative training procedure (e.g., stochastic gradient descent). This motivates the question whether information available during the training process (e.g., validation accuracy after each epoch) can be exploited for improving the epoch efficiency of BO algorithms by early-stopping model training under hyperparameter settings that will end up under-performing and hence eliminating unnecessary training epochs. This paper proposes to unify BO (specifically, Gaussian process-upper confidence bound (GP-UCB)) with Bayesian optimal stopping (BO-BOS) to boost the epoch efficiency of BO. To achieve this, while GP-UCB is sample-efficient in the number of function evaluations, BOS complements it with epoch efficiency for each function evaluation by providing a principled optimal stopping mechanism for early stopping. BO-BOS preserves the (asymptotic) no-regret performance of GP-UCB using our specified choice of BOS parameters that is amenable to an elegant interpretation in terms of the exploration-exploitation trade-off. We empirically evaluate the performance of BO-BOS and demonstrate its generality in hyperparameter optimization of ML models and two other interesting applications.

Author Information

Zhongxiang Dai (National University of Singapore)
Haibin Yu (National University of Singapore)
Bryan Kian Hsiang Low (National University of Singapore)

Dr. Bryan Low is an Associate Professor of Computer Science at the National University of Singapore and the Deputy Director of AI Research at AI Singapore. He obtained the B.Sc. (Hons.) and M.Sc. degrees in Computer Science from National University of Singapore, Singapore, in 2001 and 2002, respectively, and the Ph.D. degree in Electrical and Computer Engineering from Carnegie Mellon University, Pittsburgh, Pennsylvania, in 2009. His research interests include probabilistic & automated machine learning, planning under uncertainty, and multi-agent/robot systems. Dr. Low is the recipient of the (1) Andrew P. Sage Best Transactions Paper Award for the best paper published in all 3 of the IEEE Transactions on Systems, Man, and Cybernetics - Parts A, B, and C in 2006; (2) National University of Singapore Overseas Graduate Scholarship for Ph.D. studies in Carnegie Mellon University (CMU) in 2004-2009; (3) Singapore Computer Society Prize for Best M.Sc. Thesis in School of Computing, National University of Singapore in 2003; and (4) Faculty Teaching Excellence Award in School of Computing, National University of Singapore in 2017-2018. Dr. Low has served as a World Economic Forum’s Global Future Councils Fellow for the Council on the Future of Artificial Intelligence and Robotics from Sep 2016 to Jun 2018 and an IEEE Robotics & Automation Society (RAS) Distinguished Lecturer for the IEEE RAS Technical Committee on Multi-Robot Systems in Mar 2019. He has served as an organizing chair for the IEEE RAS Summer School on Multi-Robot Systems in Jun 2016 and the AI Summer Schools in Jul 2019 and Aug 2020. Dr. Low has also served as associate editors, area chairs and program committee members, and reviewers for premier AI (specifically, multiagent systems, AI planning, robotics, machine learning) conferences: IJCAI, AAAI, ECAI, AAMAS, ICAPS, RSS, IROS, ICRA, CoRL, NeurIPS, ICML, AISTATS, ICLR and journals: TKDE, JMLR, JAIR, MLJ, TNNLS, T-ASE, IJRR, T-RO, AURO, JFR, TOSN, JAAMAS. He was the top 5% reviewer for ICML 2019 and top 33% reviewer for ICML 2020.

Patrick Jaillet (MIT)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors