ICML 2023

Timezone: »

Poster

Stabilizing Transformer Training by Preventing Attention Entropy Collapse

Shuangfei Zhai · Tatiana Likhomanenko · Etai Littwin · Dan Busbridge · Jason Ramapuram · Yizhe Zhang · Jiatao Gu · Joshua M Susskind

Tue Jul 25 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #503

in Poster Session 2 »

Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $\sigma$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $\sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $\sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $\sigma$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer to competitive performance without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at https://github.com/apple/ml-sigma-reparam.

[ PDF] [ Poster]

Author Information

Shuangfei Zhai (Apple)

Tatiana Likhomanenko (Apple)

Etai Littwin (Apple)

Dan Busbridge (Apple)

Jason Ramapuram (Apple)

Yizhe Zhang (Machine Learning Research @ )

I am a Research scientist at Apple MLR, primarily working on Natural language processing and Machine Learning. Before joining Apple, I have been at Meta AI and Microsoft Research, working on natural language generation and NLP pre-training.