Timezone: »

 
Ranking Architectures by their Feature Extraction Capabilities
Debadeepta Dey

Fri Jul 23 09:50 AM -- 09:51 AM (PDT) @

The fundamental problem in Neural Architecture Search (NAS) is to efficiently find highperforming ones from a search space of architectures. We propose a simple but powerful method for ranking architectures FEAR in any search space. FEAR leverages the viewpoint that neural networks are powerful non-linear feature extractors. By training different architectures in the search space to the same training or validation error and subsequently comparing the usefulness of the features extracted on the task-dataset of interest by freezing most of the architecture we obtain quick estimates of the relative performance. We validate FEAR on Natsbench topology search space on three different datasets against competing baselines and show strong ranking correlation especially compared to recently proposed zero-cost methods. FEAR especially excels at ranking high-performance architectures in the search space. When used in the inner loop of discrete search algorithms like random search, FEAR can cut down the search time by approximately 2.4x without losing accuracy. We additionally empirically study very recently proposed zero-cost measures for ranking and find that they breakdown in ranking performance as training proceeds and also that data-agnostic ranking scores which ignore the dataset do not generalize across dissimilar datasets.

link to postersession #2

Author Information

Debadeepta Dey (Microsoft)

More from the Same Authors

  • 2021 : Ranking Architectures by Feature Extraction Capabilities »
    Debadeepta Dey · Shital Shah · Sebastien Bubeck
  • 2021 Poster: Boosting the Throughput and Accelerator Utilization of Specialized CNN Inference Beyond Increasing Batch Size »
    Jack Kosaian · Amar Phanishayee · Matthai Philipose · Debadeepta Dey · Rashmi Vinayak
  • 2021 Spotlight: Boosting the Throughput and Accelerator Utilization of Specialized CNN Inference Beyond Increasing Batch Size »
    Jack Kosaian · Amar Phanishayee · Matthai Philipose · Debadeepta Dey · Rashmi Vinayak
  • 2019 : Poster Session 1 (all papers) »
    Matilde Gargiani · Yochai Zur · Chaim Baskin · Evgenii Zheltonozhskii · Liam Li · Ameet Talwalkar · Xuedong Shang · Harkirat Singh Behl · Atilim Gunes Baydin · Ivo Couckuyt · Tom Dhaene · Chieh Lin · Wei Wei · Min Sun · Orchid Majumder · Michele Donini · Yoshihiko Ozaki · Ryan P. Adams · Christian Geißler · Ping Luo · zhanglin peng · · Ruimao Zhang · John Langford · Rich Caruana · Debadeepta Dey · Charles Weill · Xavi Gonzalvo · Scott Yang · Scott Yak · Eugen Hotaj · Vladimir Macko · Mehryar Mohri · Corinna Cortes · Stefan Webb · Jonathan Chen · Martin Jankowiak · Noah Goodman · Aaron Klein · Frank Hutter · Mojan Javaheripi · Mohammad Samragh · Sungbin Lim · Taesup Kim · SUNGWOONG KIM · Michael Volpp · Iddo Drori · Yamuna Krishnamurthy · Kyunghyun Cho · Stanislaw Jastrzebski · Quentin de Laroussilhe · Mingxing Tan · Xiao Ma · Neil Houlsby · Andrea Gesmundo · Zalán Borsos · Krzysztof Maziarz · Felipe Petroski Such · Joel Lehman · Kenneth Stanley · Jeff Clune · Pieter Gijsbers · Joaquin Vanschoren · Felix Mohr · Eyke Hüllermeier · Zheng Xiong · Wenpeng Zhang · Wenwu Zhu · Weijia Shao · Aleksandra Faust · Michal Valko · Michael Y Li · Hugo Jair Escalante · Marcel Wever · Andrey Khorlin · Tara Javidi · Anthony Francis · Saurajit Mukherjee · Jungtaek Kim · Michael McCourt · Saehoon Kim · Tackgeun You · Seungjin Choi · Nicolas Knudde · Alexander Tornede · Ghassen Jerfel
  • 2017 Poster: Safety-Aware Algorithms for Adversarial Contextual Bandit »
    Wen Sun · Debadeepta Dey · Ashish Kapoor
  • 2017 Talk: Safety-Aware Algorithms for Adversarial Contextual Bandit »
    Wen Sun · Debadeepta Dey · Ashish Kapoor