Timezone: »
Recent advances in neural architecture search (NAS) demand tremendous computational resources. This makes it difficult to reproduce experiments and imposes a barrier to entry to researchers without access to large scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build it, we carefully constructed a compact---yet expressive---search space, exploiting graph isomorphisms to identify 423K unique architectures. Utilizing machine-years of computation, we trained them all with public code, and compiled the results into a large table. This allows researchers to evaluate the quality of a proposed model in milliseconds using various precomputed metrics. NAS-Bench-101 presents a unique opportunity to study the entire NAS loss landscape from a data-driven perspective, which we illustrate with our analysis. We also demonstrate the dataset's application to benchmarking by comparing a range of popular architecture optimization algorithms on it.
Author Information
Chris Ying (Ambient.ai)
Aaron Klein (University of Freiburg)
Eric Christiansen (Google)
Esteban Real (Google Inc.)
Kevin Murphy (Google Brain)
Frank Hutter (University of Freiburg and Bosch Center for Artificial Intelligence)
Frank Hutter is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), where he has been a faculty member since 2013. Before that, he was at the University of British Columbia (UBC) for eight years, for his PhD and postdoc. Frank's main research interests lie in machine learning, artificial intelligence and automated algorithm design. For his 2009 PhD thesis on algorithm configuration, he received the CAIAC doctoral dissertation award for the best thesis in AI in Canada that year, and with his coauthors, he received several best paper awards and prizes in international competitions on automated machine learning, SAT solving, and AI planning. Since 2016 he holds an ERC Starting Grant for a project on automating deep learning based on Bayesian optimization, Bayesian neural networks, and deep reinforcement learning.
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: NAS-Bench-101: Towards Reproducible Neural Architecture Search »
Thu Jun 13th 01:30 -- 04:00 AM Room Pacific Ballroom
More from the Same Authors
-
2020 Workshop: 7th ICML Workshop on Automated Machine Learning (AutoML 2020) »
Frank Hutter · Joaquin Vanschoren · Marius Lindauer · Charles Weill · Katharina Eggensperger · Matthias Feurer · Matthias Feurer -
2020 Poster: Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems »
Zhe Dong · Bryan Seybold · Kevin Murphy · Hung Bui -
2020 Poster: Population-Based Black-Box Optimization for Biological Sequence Design »
Christof Angermueller · David Belanger · Andreea Gane · Zelda Mariet · David Dohan · Kevin Murphy · Lucy Colwell · D. Sculley -
2020 Poster: AutoML-Zero: Evolving Machine Learning Algorithms From Scratch »
Esteban Real · Chen Liang · David So · Quoc Le -
2019 Workshop: 6th ICML Workshop on Automated Machine Learning (AutoML 2019) »
Frank Hutter · Joaquin Vanschoren · Katharina Eggensperger · Matthias Feurer · Matthias Feurer -
2019 Tutorial: Algorithm configuration: learning in the space of algorithm designs »
Kevin Leyton-Brown · Frank Hutter -
2018 Poster: BOHB: Robust and Efficient Hyperparameter Optimization at Scale »
Stefan Falkner · Aaron Klein · Frank Hutter -
2018 Poster: Fixing a Broken ELBO »
Alexander Alemi · Ben Poole · Ian Fischer · Joshua V Dillon · Rif Saurous · Kevin Murphy -
2018 Oral: Fixing a Broken ELBO »
Alexander Alemi · Ben Poole · Ian Fischer · Joshua V Dillon · Rif Saurous · Kevin Murphy -
2018 Oral: BOHB: Robust and Efficient Hyperparameter Optimization at Scale »
Stefan Falkner · Aaron Klein · Frank Hutter -
2017 Poster: Large-Scale Evolution of Image Classifiers »
Esteban Real · Sherry Moore · Andrew Selle · Saurabh Saxena · Yutaka Leon Suematsu · Jie Tan · Quoc Le · Alexey Kurakin -
2017 Talk: Large-Scale Evolution of Image Classifiers »
Esteban Real · Sherry Moore · Andrew Selle · Saurabh Saxena · Yutaka Leon Suematsu · Jie Tan · Quoc Le · Alexey Kurakin