Timezone: »

 
Oral
Fast Decoding in Sequence Models Using Discrete Latent Variables
Lukasz Kaiser · Samy Bengio · Aurko Roy · Ashish Vaswani · Niki Parmar · Jakob Uszkoreit · Noam Shazeer

Thu Jul 12 06:00 AM -- 06:10 AM (PDT) @ Victoria
in Deep Learning (Neural Network Architectures) 8 »

Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and Transformer are the state-of-the-art on many tasks. However, they lack parallelism and are thus slow for long sequences. RNNs lack parallelism both during training and decoding, while architectures like WaveNet and Transformer are much more parallel during training, but still lack parallelism during decoding.We present a method to extend sequence models using discrete latent variables that makes decoding much more parallel. The main idea behind this approach is to first autoencode the target sequence into a shorter discrete latent sequence, which is generated autoregressively, and finally decode the full sequence from this shorter latent sequence in a parallel manner. To this end, we introduce a new method for constructing discrete latent variables and compare it with previously introduced methods. Finally, we verify that our model works on the task of neural machine translation, where our models are an order of magnitude faster than comparable autoregressive models and, while lower in BLEU than purely autoregressive models, better than previously proposed non-autogregressive translation.

[ PDF] 

Author Information

Lukasz Kaiser (Google)
Samy Bengio (Google Brain)
Aurko Roy (Google Brain)
Ashish Vaswani (Google Brain)
Niki Parmar (Google)
Jakob Uszkoreit
Noam Shazeer (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors