NAS-Bench-101: Towards Reproducible Neural Architecture Search

Recent advances in neural architecture search (NAS) demand tremendous computational resources. This makes it difficult to reproduce experiments and imposes a barrier to entry to researchers without access to large scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build it, we carefully constructed a compact---yet expressive---search space, exploiting graph isomorphisms to identify 423K unique architectures. Utilizing machine-years of computation, we trained them all with public code, and compiled the results into a large table. This allows researchers to evaluate the quality of a proposed model in milliseconds using various precomputed metrics. NAS-Bench-101 presents a unique opportunity to study the entire NAS loss landscape from a data-driven perspective, which we illustrate with our analysis. We also demonstrate the dataset's application to benchmarking by comparing a range of popular architecture optimization algorithms on it.