Timezone: »

 
Spotlight
SparseBERT: Rethinking the Importance Analysis in Self-attention
Han Shi · Jiahui Gao · Xiaozhe Ren · Hang Xu · Xiaodan Liang · Zhenguo Li · James Kwok

Thu Jul 22 06:20 AM -- 06:25 AM (PDT) @
in Applications (NLP) 1 »

Transformer-based models are popularly used in natural language processing (NLP). Its core component, self-attention, has aroused widespread interest. To understand the self-attention mechanism, a direct method is to visualize the attention map of a pre-trained model. Based on the patterns observed, a series of efficient Transformers with different sparse attention masks have been proposed. From a theoretical perspective, universal approximability of Transformer-based models is also recently proved. However, the above understanding and analysis of self-attention is based on a pre-trained model. To rethink the importance analysis in self-attention, we study the significance of different positions in attention matrix during pre-training. A surprising result is that diagonal elements in the attention map are the least important compared with other attention positions. We provide a proof showing that these diagonal elements can indeed be removed without deteriorating model performance. Furthermore, we propose a Differentiable Attention Mask (DAM) algorithm, which further guides the design of the SparseBERT. Extensive experiments verify our interesting findings and illustrate the effect of the proposed algorithm.

Keywords: Natural Language Processing

Author Information

Han Shi (The Hong Kong University of Science and Technology)
Jiahui Gao (The University of Hong Kong)
Xiaozhe Ren (Huawei)
Hang Xu (Huawei Noah's Ark Lab)
Xiaodan Liang (Sun Yat-sen University)
Zhenguo Li (Huawei Tech. Investment, Co., Ltd)
James Kwok (Hong Kong University of Science and Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors