Timezone: »

 
Poster
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
Weizhen Qi · Yeyun Gong · Jian Jiao · Yu Yan · Weizhu Chen · Dayiheng Liu · Kewen Tang · Houqiang Li · Jiusheng Chen · Ruofei Zhang · Ming Zhou · Nan Duan

Thu Jul 22 09:00 PM -- 11:00 PM (PDT) @
in Poster Session 6 »

In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation through designing a novel model structure for large-scale pre-training. A pretrained BANG model can simultaneously support AR, NAR, and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum), and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39, and 5.90 in the overall scores of SQuAD, XSUM, and PersonaChat compared with the NAR strong baselines, respectively. Our code will be made publicly available.

Keywords: Natural Language Processing

Author Information

Weizhen Qi (University of Science and Technology of China)
Yeyun Gong (Microsoft Research Asia)
Jian Jiao (Microsoft)
Yu Yan (Microsoft)
Weizhu Chen (Microsoft)
Dayiheng Liu (Alibaba DAMO Academy)
Kewen Tang (Microsoft)
Houqiang Li (University of Science and Technology of China)
Jiusheng Chen (Microsoft)
Ruofei Zhang (Microsoft)
Ming Zhou (Microsoft Research)
Nan Duan (Microsoft Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors