Timezone: »
Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.
Author Information
Xiangning Chen (UCLA)
Cho-Jui Hsieh (UCLA)
More from the Same Authors
-
2021 : Fast Certified Robust Training with Short Warmup »
Zhouxing Shi · Yihan Wang · Huan Zhang · Jinfeng Yi · Cho-Jui Hsieh -
2021 : Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification »
Shiqi Wang · Huan Zhang · Kaidi Xu · Xue Lin · Suman Jana · Cho-Jui Hsieh · Zico Kolter -
2023 : Formal Verification for Neural Networks with General Nonlinearities via Branch-and-Bound »
Zhouxing Shi · Qirui Jin · Huan Zhang · Zico Kolter · Suman Jana · Cho-Jui Hsieh -
2023 Workshop: 2nd Workshop on Formal Verification of Machine Learning »
Mark Müller · Brendon G. Anderson · Leslie Rice · Zhouxing Shi · Shubham Ugare · Huan Zhang · Martin Vechev · Zico Kolter · Somayeh Sojoudi · Cho-Jui Hsieh -
2023 Poster: Representer Point Selection for Explaining Regularized High-dimensional Models »
Che-Ping Tsai · Jiong Zhang · Hsiang-Fu Yu · Eli Chien · Cho-Jui Hsieh · Pradeep Ravikumar -
2023 Poster: PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation »
Eli Chien · Jiong Zhang · Cho-Jui Hsieh · Jyun-Yu Jiang · Wei-Cheng Chang · Olgica Milenkovic · Hsiang-Fu Yu -
2023 Poster: Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory »
Justin Cui · Ruochen Wang · Si Si · Cho-Jui Hsieh -
2022 Workshop: Workshop on Formal Verification of Machine Learning »
Huan Zhang · Leslie Rice · Kaidi Xu · aditi raghunathan · Wan-Yi Lin · Cho-Jui Hsieh · Clark Barrett · Martin Vechev · Zico Kolter -
2022 Poster: A Branch and Bound Framework for Stronger Adversarial Attacks of ReLU Networks »
Huan Zhang · Shiqi Wang · Kaidi Xu · Yihan Wang · Suman Jana · Cho-Jui Hsieh · Zico Kolter -
2022 Spotlight: A Branch and Bound Framework for Stronger Adversarial Attacks of ReLU Networks »
Huan Zhang · Shiqi Wang · Kaidi Xu · Yihan Wang · Suman Jana · Cho-Jui Hsieh · Zico Kolter -
2021 Poster: Overcoming Catastrophic Forgetting by Bayesian Generative Regularization »
PEI-HUNG Chen · Wei Wei · Cho-Jui Hsieh · Bo Dai -
2021 Spotlight: Overcoming Catastrophic Forgetting by Bayesian Generative Regularization »
PEI-HUNG Chen · Wei Wei · Cho-Jui Hsieh · Bo Dai -
2020 Poster: On Lp-norm Robustness of Ensemble Decision Stumps and Trees »
Yihan Wang · Huan Zhang · Hongge Chen · Duane Boning · Cho-Jui Hsieh -
2020 Poster: Learning to Encode Position for Transformer with Continuous Dynamical Model »
Xuanqing Liu · Hsiang-Fu Yu · Inderjit Dhillon · Cho-Jui Hsieh -
2019 Poster: Robust Decision Trees Against Adversarial Examples »
Hongge Chen · Huan Zhang · Duane Boning · Cho-Jui Hsieh -
2019 Oral: Robust Decision Trees Against Adversarial Examples »
Hongge Chen · Huan Zhang · Duane Boning · Cho-Jui Hsieh