Timezone: »

 
Robustness in deep learning: The width (good), the depth (bad), and the initialization (ugly)
Zhenyu Zhu · Fanghui Liu · Grigorios Chrysos · Volkan Cevher

@
in New Frontiers in Adversarial Machine Learning »

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth exacerbates the robustness. Moreover, under non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al. NeurIPS21; Wu et al. NeurIPS21] and are consistent with [Bubeck and Sellke NeurIPS21; Bubeck et al. COLT21].

Author Information

Zhenyu Zhu (EPFL)
Fanghui Liu (EPFL)

l am currently a postdoc researcher in EPFL, and my research interest includes statistical machine learning, mainly on kernel methods and learning theory.

Grigorios Chrysos (EPFL)
Volkan Cevher (EPFL)

More from the Same Authors