Timezone: »

 
Spotlight
High-Performance Large-Scale Image Recognition Without Normalization
Andy Brock · Soham De · Samuel Smith · Karen Simonyan

Tue Jul 20 07:40 PM -- 07:45 PM (PDT) @
in AutoML and Neural Network Architectures 2 »

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when fine-tuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%.

Keywords: Deep Learning

Author Information

Andy Brock (DeepMind)
Soham De (DeepMind)
Samuel Smith (DeepMind)
Karen Simonyan (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors