Timezone: »

OmniNet: Omnidirectional Representations from Transformers
Yi Tay · Mostafa Dehghani · Vamsi Aribandi · Jai Gupta · Philip Pham · Zhen Qin · Dara Bahri · Da-Cheng Juan · Don Metzler

Tue Jul 20 09:00 AM -- 11:00 AM (PDT) @ None #None

This paper proposes Omnidirectional Representations from Transformers (OMNINET). In OmniNet, instead of maintaining a strictly horizon-tal receptive field, each token is allowed to attend to all tokens in the entire network. This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based, low-rank attention and/or Big Bird as the meta-learner. Extensive experiments are conducted on autoregressive language modeling(LM1B, C4), Machine Translation, Long Range Arena (LRA), and Image Recognition.The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B,WMT’14 En-De/En-Fr, and Long Range Arena.Moreover, using omnidirectional representation in Vision Transformers leads to significant improvements on image recognition tasks on both few-shot learning and fine-tuning setups.

Author Information

Yi Tay (Google)
Mostafa Dehghani (Google Brain)
Vamsi Aribandi (BITS Pilani - Hyderabad)
Jai Gupta (Google)
Philip Pham (Google Research)
Zhen Qin (Google)
Dara Bahri (Google Research)
Da-Cheng Juan (Google)
Don Metzler (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors