Timezone: »
Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.
Author Information
Colin Raffel (Google Brain)
Thang Luong (Google Brain)
Peter Liu (Google Brain)
Ron Weiss (Google Brain)
Douglas Eck (Google Brain)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Talk: Online and Linear-Time Attention by Enforcing Monotonic Alignments »
Tue. Aug 8th 03:30 -- 03:48 AM Room Parkside 1
More from the Same Authors
-
2023 Poster: Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models »
Nikhil Kandpal · Brian Lester · Mohammed Muqeeth · Anisha Mascarenhas · Monty Evans · Vishal Baskaran · Tenghao Huang · Haokun Liu · Colin Raffel -
2023 Poster: Large Language Models Struggle to Learn Long-Tail Knowledge »
Nikhil Kandpal · Haikang Deng · Adam Roberts · Eric Wallace · Colin Raffel -
2022 Workshop: The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward »
Huaxiu Yao · Hugo Larochelle · Percy Liang · Colin Raffel · Jian Tang · Ying WEI · Saining Xie · Eric Xing · Chelsea Finn -
2022 Poster: Deduplicating Training Data Mitigates Privacy Risks in Language Models »
Nikhil Kandpal · Eric Wallace · Colin Raffel -
2022 Spotlight: Deduplicating Training Data Mitigates Privacy Risks in Language Models »
Nikhil Kandpal · Eric Wallace · Colin Raffel -
2022 Poster: What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? »
Thomas Wang · Adam Roberts · Daniel Hesslow · Teven Le Scao · Hyung Won Chung · Iz Beltagy · Julien Launay · Colin Raffel -
2022 Spotlight: What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? »
Thomas Wang · Adam Roberts · Daniel Hesslow · Teven Le Scao · Hyung Won Chung · Iz Beltagy · Julien Launay · Colin Raffel -
2021 Poster: Emergent Social Learning via Multi-agent Reinforcement Learning »
Kamal Ndousse · Douglas Eck · Sergey Levine · Natasha Jaques -
2021 Poster: Towards Domain-Agnostic Contrastive Learning »
Vikas Verma · Thang Luong · Kenji Kawaguchi · Hieu Pham · Quoc Le -
2021 Spotlight: Emergent Social Learning via Multi-agent Reinforcement Learning »
Kamal Ndousse · Douglas Eck · Sergey Levine · Natasha Jaques -
2021 Spotlight: Towards Domain-Agnostic Contrastive Learning »
Vikas Verma · Thang Luong · Kenji Kawaguchi · Hieu Pham · Quoc Le -
2019 Poster: Learning to Groove with Inverse Sequence Transformations »
Jon Gillick · Adam Roberts · Jesse Engel · Douglas Eck · David Bamman -
2019 Oral: Learning to Groove with Inverse Sequence Transformations »
Jon Gillick · Adam Roberts · Jesse Engel · Douglas Eck · David Bamman -
2018 Poster: Learning Longer-term Dependencies in RNNs with Auxiliary Losses »
Trieu H Trinh · Andrew Dai · Thang Luong · Quoc Le -
2018 Oral: Learning Longer-term Dependencies in RNNs with Auxiliary Losses »
Trieu H Trinh · Andrew Dai · Thang Luong · Quoc Le -
2018 Poster: Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron »
RJ Skerry-Ryan · Eric Battenberg · Ying Xiao · Yuxuan Wang · Daisy Stanton · Joel Shor · Ron Weiss · Robert Clark · Rif Saurous -
2018 Poster: A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music »
Adam Roberts · Jesse Engel · Colin Raffel · Curtis Hawthorne · Douglas Eck -
2018 Oral: Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron »
RJ Skerry-Ryan · Eric Battenberg · Ying Xiao · Yuxuan Wang · Daisy Stanton · Joel Shor · Ron Weiss · Robert Clark · Rif Saurous -
2018 Oral: A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music »
Adam Roberts · Jesse Engel · Colin Raffel · Curtis Hawthorne · Douglas Eck -
2017 Poster: Sequence Tutor: Conservative fine-tuning of sequence generation models with KL-control »
Natasha Jaques · Shixiang Gu · Dzmitry Bahdanau · Jose Miguel Hernandez-Lobato · Richard E Turner · Douglas Eck -
2017 Poster: Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders »
Cinjon Resnick · Adam Roberts · Jesse Engel · Douglas Eck · Sander Dieleman · Karen Simonyan · Mohammad Norouzi -
2017 Talk: Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders »
Cinjon Resnick · Adam Roberts · Jesse Engel · Douglas Eck · Sander Dieleman · Karen Simonyan · Mohammad Norouzi -
2017 Talk: Sequence Tutor: Conservative fine-tuning of sequence generation models with KL-control »
Natasha Jaques · Shixiang Gu · Dzmitry Bahdanau · Jose Miguel Hernandez-Lobato · Richard E Turner · Douglas Eck