Timezone: »

 
Spotlight
Examining Scaling and Transfer of Language Model Architectures for Machine Translation
Biao Zhang · Behrooz Ghorbani · Ankur Bapna · Yong Cheng · Xavier Garcia · Jonathan Shen · Orhan Firat

Wed Jul 20 08:50 AM -- 08:55 AM (PDT) @ Hall G
in Applications »

Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, and improve greatly on zero-shot directions by facilitating the reduction of off-target translations.

Keywords: APP: Language, Speech and Dialog

Author Information

Biao Zhang (University of Edinburgh)

Biao Zhang is a final-year Ph.D. student at the ILCC at the University of Edinburgh under the supervision of Prof. Rico Sennrich and Prof. Ivan Titov. His research focuses on improving neural machine translation (NMT), particularly its efficiency and universality, including developing lightweight (fast and effective) architectures for NMT, low-resource NMT, massively multilingual NMT, speech-to-text translation, context-aware NMT, and their intersections.

Behrooz Ghorbani (Google Research)
Ankur Bapna (Google Research)
Yong Cheng (Google)
Xavier Garcia (Google)
Jonathan Shen (Independent)
Orhan Firat (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors