Decoupled Differentiable Neural Architecture Search: Memory-Efficient Differentiable NAS via Disentangled Search Space

Differentiable Neural Architecture Search (NAS) is a popular paradigm, but scaling this approach to models with larger parameters is severely hampered by the fact that the entire supernet resides in GPU memory. In this paper, we rethink the gradient propagation processs of Differentiable NAS and propose Decoupled Differentiable Neural Architecture Search (D2NAS). In our method, the branch structure is designed to decouple the weight update of the trainable parameters from the backbone network, and the candidate operation selection is redesigned with Gumbel-Softmax to make the overall differentiable process more stable. Experiments show that D2NAS achieves both performance and stability, with 67\% memory cost compared to the best other differentiable methods.