Timezone: »

 
Spotlight
Wide Neural Networks Forget Less Catastrophically
Seyed Iman Mirzadeh · Arslan Chaudhry · Dong Yin · Huiyi Hu · Razvan Pascanu · Dilan Gorur · Mehrdad Farajtabar

Thu Jul 21 11:35 AM -- 11:40 AM (PDT) @ Room 310
in Probabilistic Methods/MISC »

A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent progress in continual learning literature is encouraging, our understanding of what properties of neural networks contribute to catastrophic forgetting is still limited. To address this, instead of focusing on continual learning algorithms, in this work, we focus on the model itself and study the impact of "width" of the neural network architecture on catastrophic forgetting, and show that width has a surprisingly significant effect on forgetting. To explain this effect, we study the learning dynamics of the network from various perspectives such as gradient orthogonality, sparsity, and lazy training regime. We provide potential explanations that are consistent with the empirical results across different architectures and continual learning benchmarks.

Keywords: MISC: Transfer, Multitask and Meta-learning

Author Information

Seyed Iman Mirzadeh (Washington State University)
Arslan Chaudhry (DeepMind)

Arslan is a Research Scientist at DeepMind in Mountain View. He is interested in Machine Learning models that can learn efficiently from multiple tasks. Towards this, he studies continual, meta- and transfer learning. He did his PhD in Machine Learning with Prof. Philip H. S. Torr at the University of Oxford. He was funded by the Rhodes Trust. Before joining Oxford, he worked as a software developer at Mentor Graphics where he helped develop Mentor Embedded Hypervisor. Prior to that, he did his undergraduate in Electrical Engineering from the University of Engineering and Technology, Lahore.

Dong Yin (DeepMind)
Huiyi Hu (DeepMind)
Razvan Pascanu (DeepMind)
Dilan Gorur
Mehrdad Farajtabar (Google DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors