Timezone: »

 
Poster
KNAS: Green Neural Architecture Search
Jingjing Xu · Liang Zhao · Junyang Lin · Rundong Gao · Xu SUN · Hongxia Yang

Tue Jul 20 09:00 AM -- 11:00 AM (PDT) @ None #None
in Poster Session 1 »

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than ``train-then-test'' paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks.

Keywords: Architectures

Author Information

Jingjing Xu (ByteDance AI Lab)
Liang Zhao (Peking Unviersity)
Junyang Lin (Alibaba Group)
Rundong Gao (Tsinghua University)
Xu SUN (Peking University)
Hongxia Yang (Alibaba Group)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors