Timezone: »

 
Spotlight
Flowformer: Linearizing Transformers with Conservation Flows
Haixu Wu · Jialong Wu · Jiehui Xu · Jianmin Wang · Mingsheng Long

Thu Jul 21 01:30 PM -- 01:35 PM (PDT) @ Ballroom 1 & 2
in Deep Learning »

Transformers based on the attention mechanism have achieved impressive success in various areas. However, the attention mechanism has a quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. Previous methods mainly utilize the similarity decomposition and the associativity of matrix multiplication to devise linear-time attention mechanisms. They avoid degeneration of attention to a trivial distribution by reintroducing inductive biases such as the locality, thereby at the expense of model generality and expressiveness. In this paper, we linearize Transformers free from specific inductive biases based on the flow network theory. We cast attention as the information flow aggregated from the sources (values) to the sinks (results) through the learned flow capacities (attentions). Within this framework, we apply the property of flow conservation into attention and propose the Flow-Attention mechanism of linear complexity. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions without using specific inductive biases. Empowered by the Flow-Attention, Flowformer yields strong performance in linear time for wide areas, including long sequence, time series, vision, natural language, and reinforcement learning. The code and settings are available at this repository: https://github.com/thuml/Flowformer.

Keywords: DL: Attention Mechanisms

Author Information

Haixu Wu (Tsinghua University)
Jialong Wu (Tsinghua University)
Jiehui Xu (THU)
Jianmin Wang (Tsinghua University)
Mingsheng Long (Tsinghua University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors