Timezone: »
Recent advances in neural architecture search (NAS) demand tremendous computational resources. This makes it difficult to reproduce experiments and imposes a barrier to entry to researchers without access to large scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build it, we carefully constructed a compact---yet expressive---search space, exploiting graph isomorphisms to identify 423K unique architectures. Utilizing machine-years of computation, we trained them all with public code, and compiled the results into a large table. This allows researchers to evaluate the quality of a proposed model in milliseconds using various precomputed metrics. NAS-Bench-101 presents a unique opportunity to study the entire NAS loss landscape from a data-driven perspective, which we illustrate with our analysis. We also demonstrate the dataset's application to benchmarking by comparing a range of popular architecture optimization algorithms on it.
Author Information
Chris Ying (Ambient.ai)
Aaron Klein (University of Freiburg)
Eric Christiansen (Google)
Esteban Real (Google Inc.)
Kevin Murphy (Google Brain)
Frank Hutter (University of Freiburg and Bosch Center for Artificial Intelligence)
Frank Hutter is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), where he has been a faculty member since 2013. Before that, he was at the University of British Columbia (UBC) for eight years, for his PhD and postdoc. Frank's main research interests lie in machine learning, artificial intelligence and automated algorithm design. For his 2009 PhD thesis on algorithm configuration, he received the CAIAC doctoral dissertation award for the best thesis in AI in Canada that year, and with his coauthors, he received several best paper awards and prizes in international competitions on automated machine learning, SAT solving, and AI planning. Since 2016 he holds an ERC Starting Grant for a project on automating deep learning based on Bayesian optimization, Bayesian neural networks, and deep reinforcement learning.
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: NAS-Bench-101: Towards Reproducible Neural Architecture Search »
Thu. Jun 13th 01:30 -- 04:00 AM Room Pacific Ballroom #12
More from the Same Authors
-
2021 : Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise Ratio »
Julien N Siems · Aaron Klein · Cedric Archambeau · Maren Mahsereci -
2021 : Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization »
Sergio Izquierdo · Julia Guerrero-Viu · Sven Hauns · Guilherme Miotto · Simon Schrodi · André Biedenkapp · Thomas Elsken · Difan Deng · Marius Lindauer · Frank Hutter -
2022 : P30: Meta-Learning Real-Time Bayesian AutoML For Small Tabular Data »
Frank Hutter · Katharina Eggensperger -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · Jie Ren · Joost van Amersfoort · Kehang Han · E. Kelly Buchanan · Kevin Murphy · Mark Collier · Mike Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2022 : On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning »
Diane Wagner · Fabio Ferreira · Danny Stoll · Robin Tibor Schirrmeister · Samuel Gabriel Müller · Frank Hutter -
2023 : CAAFE: Combining Large Language Models with Tabular Predictors for Semi-Automated Data Science »
Noah Hollmann · Samuel Gabriel Müller · Frank Hutter -
2023 : Guided Evolution with Binary Predictors for ML Program Search »
John Co-Reyes · Yingjie Miao · George Tucker · Aleksandra Faust · Esteban Real -
2023 Poster: PFNs4BO: In-Context Learning for Bayesian Optimization »
Samuel Gabriel Müller · Matthias Feurer · Noah Hollmann · Frank Hutter -
2023 Poster: Optimizing Hyperparameters with Conformal Quantile Regression »
David Salinas · Jacek Golebiowski · Aaron Klein · Matthias Seeger · Cedric Archambeau -
2022 Poster: Zero-shot AutoML with Pretrained Models »
Ekrem Öztürk · Fabio Ferreira · Hadi S Jomaa · Lars Schmidt-Thieme · Josif Grabocka · Frank Hutter -
2022 Spotlight: Zero-shot AutoML with Pretrained Models »
Ekrem Öztürk · Fabio Ferreira · Hadi S Jomaa · Lars Schmidt-Thieme · Josif Grabocka · Frank Hutter -
2021 Workshop: 8th ICML Workshop on Automated Machine Learning (AutoML 2021) »
Gresa Shala · Frank Hutter · Joaquin Vanschoren · Marius Lindauer · Katharina Eggensperger · Colin White · Erin LeDell -
2021 Poster: Self-Paced Context Evaluation for Contextual Reinforcement Learning »
Theresa Eimer · André Biedenkapp · Frank Hutter · Marius Lindauer -
2021 Poster: TempoRL: Learning When to Act »
André Biedenkapp · Raghu Rajan · Frank Hutter · Marius Lindauer -
2021 Spotlight: TempoRL: Learning When to Act »
André Biedenkapp · Raghu Rajan · Frank Hutter · Marius Lindauer -
2021 Spotlight: Self-Paced Context Evaluation for Contextual Reinforcement Learning »
Theresa Eimer · André Biedenkapp · Frank Hutter · Marius Lindauer -
2021 Poster: BORE: Bayesian Optimization by Density-Ratio Estimation »
Louis Chi-Chun Tiao · Aaron Klein · Matthias W Seeger · Edwin V. Bonilla · Cedric Archambeau · Fabio Ramos -
2021 Oral: BORE: Bayesian Optimization by Density-Ratio Estimation »
Louis Chi-Chun Tiao · Aaron Klein · Matthias W Seeger · Edwin V. Bonilla · Cedric Archambeau · Fabio Ramos -
2020 Workshop: 7th ICML Workshop on Automated Machine Learning (AutoML 2020) »
Frank Hutter · Joaquin Vanschoren · Marius Lindauer · Charles Weill · Katharina Eggensperger · Matthias Feurer · Matthias Feurer -
2020 : Welcome »
Frank Hutter -
2020 Poster: Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems »
Zhe Dong · Bryan Seybold · Kevin Murphy · Hung Bui -
2020 Poster: Population-Based Black-Box Optimization for Biological Sequence Design »
Christof Angermueller · David Belanger · Andreea Gane · Zelda Mariet · David Dohan · Kevin Murphy · Lucy Colwell · D. Sculley -
2020 Poster: AutoML-Zero: Evolving Machine Learning Algorithms From Scratch »
Esteban Real · Chen Liang · David So · Quoc Le -
2019 : Closing Remarks »
Frank Hutter -
2019 : Poster Session 1 (all papers) »
Matilde Gargiani · Yochai Zur · Chaim Baskin · Evgenii Zheltonozhskii · Liam Li · Ameet Talwalkar · Xuedong Shang · Harkirat Singh Behl · Atilim Gunes Baydin · Ivo Couckuyt · Tom Dhaene · Chieh Lin · Wei Wei · Min Sun · Orchid Majumder · Michele Donini · Yoshihiko Ozaki · Ryan P. Adams · Christian Geißler · Ping Luo · zhanglin peng · · Ruimao Zhang · John Langford · Rich Caruana · Debadeepta Dey · Charles Weill · Xavi Gonzalvo · Scott Yang · Scott Yak · Eugen Hotaj · Vladimir Macko · Mehryar Mohri · Corinna Cortes · Stefan Webb · Jonathan Chen · Martin Jankowiak · Noah Goodman · Aaron Klein · Frank Hutter · Mojan Javaheripi · Mohammad Samragh · Sungbin Lim · Taesup Kim · SUNGWOONG KIM · Michael Volpp · Iddo Drori · Yamuna Krishnamurthy · Kyunghyun Cho · Stanislaw Jastrzebski · Quentin de Laroussilhe · Mingxing Tan · Xiao Ma · Neil Houlsby · Andrea Gesmundo · Zalán Borsos · Krzysztof Maziarz · Felipe Petroski Such · Joel Lehman · Kenneth Stanley · Jeff Clune · Pieter Gijsbers · Joaquin Vanschoren · Felix Mohr · Eyke Hüllermeier · Zheng Xiong · Wenpeng Zhang · Wenwu Zhu · Weijia Shao · Aleksandra Faust · Michal Valko · Michael Y Li · Hugo Jair Escalante · Marcel Wever · Andrey Khorlin · Tara Javidi · Anthony Francis · Saurajit Mukherjee · Jungtaek Kim · Michael McCourt · Saehoon Kim · Tackgeun You · Seungjin Choi · Nicolas Knudde · Alexander Tornede · Ghassen Jerfel -
2019 : Welcome »
Frank Hutter -
2019 Workshop: 6th ICML Workshop on Automated Machine Learning (AutoML 2019) »
Frank Hutter · Joaquin Vanschoren · Katharina Eggensperger · Matthias Feurer · Matthias Feurer -
2019 Tutorial: Algorithm configuration: learning in the space of algorithm designs »
Kevin Leyton-Brown · Frank Hutter -
2018 Poster: BOHB: Robust and Efficient Hyperparameter Optimization at Scale »
Stefan Falkner · Aaron Klein · Frank Hutter -
2018 Poster: Fixing a Broken ELBO »
Alexander Alemi · Ben Poole · Ian Fischer · Joshua V Dillon · Rif Saurous · Kevin Murphy -
2018 Oral: Fixing a Broken ELBO »
Alexander Alemi · Ben Poole · Ian Fischer · Joshua V Dillon · Rif Saurous · Kevin Murphy -
2018 Oral: BOHB: Robust and Efficient Hyperparameter Optimization at Scale »
Stefan Falkner · Aaron Klein · Frank Hutter -
2017 Poster: Large-Scale Evolution of Image Classifiers »
Esteban Real · Sherry Moore · Andrew Selle · Saurabh Saxena · Yutaka Leon Suematsu · Jie Tan · Quoc Le · Alexey Kurakin -
2017 Talk: Large-Scale Evolution of Image Classifiers »
Esteban Real · Sherry Moore · Andrew Selle · Saurabh Saxena · Yutaka Leon Suematsu · Jie Tan · Quoc Le · Alexey Kurakin