Timezone: »

 
Spotlight
Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
Vu Nguyen · Tam Le · Makoto Yamada · Michael A Osborne

Tue Jul 20 06:10 PM -- 06:15 PM (PDT) @
in AutoML »

Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), which is a negative definite variant of OT, we develop a novel discrepancy for neural architectures, and demonstrate it within a Gaussian process surrogate model for the sequential NAS settings. Furthermore, we derive a novel parallel NAS, using quality k-determinantal point process on the GP posterior, to select diverse and high-performing architectures from a discrete set of candidates. Empirically, we demonstrate that our TW-based approaches outperform other baselines in both sequential and parallel NAS.

Keywords: Algorithms · AutoML

Author Information

Vu Nguyen (Amazon Adelaide)
Tam Le (RIKEN AIP)

My name is Tam Le. I have been an assistant professor at The Institute of Statistical Mathematics (ISM), Japan since 09/2022. I am also a visiting scientist at RIKEN AIP, Japan since 12/2022. I officially received my PhD degree from Kyoto University in 01/2016, under the supervision of Professor Marco Cuturi and Professor Akihiro Yamamoto. Before ISM, I worked at RIKEN AIP as a postdoc (09/2017 - 07/2021), and as a research scientist (08/2021 - 08/2022), working with Professor Makoto Yamada. Before those, I spent 1.5 year as a postdoc at Nagoya Institute of Technology and National Institute of Materials Science (02/2016 - 08/2017), working with Professor Ichiro Takeuchi.

Makoto Yamada (RIKEN)
Michael A Osborne (U Oxford)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors