Timezone: »

 
Spotlight
Poolingformer: Long Document Modeling with Pooling Attention
Hang ZHANG · Yeyun Gong · Yelong Shen · Weisheng Li · Jiancheng Lv · Nan Duan · Weizhu Chen

Tue Jul 20 05:45 PM -- 05:50 PM (PDT) @
in Deep Learning Algorithms 6 »

In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information from neighbors. Its second level employs a larger window to increase receptive fields with pooling attention to reduce both computational cost and memory consumption. We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA. Experimental results show that Poolingformer sits atop three official leaderboards measured by F1, outperforming previous state-of-the-art models by 1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal answer. We further evaluate Poolingformer on a long sequence summarization task. Experimental results on the arXiv benchmark continue to demonstrate its superior performance.

Keywords: Natural Language Processing

Author Information

Hang ZHANG (College of Computer Science, Sichuan University)
Yeyun Gong (Microsoft Research Asia)
Yelong Shen (microsoft)
Weisheng Li (University of Science and Technology of China)
Jiancheng Lv (Sichuan University)
Nan Duan (Microsoft Research)
Weizhu Chen (Microsoft)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors