Timezone: »

 
Spotlight
General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne · Drew Jaegle · Cătălina Cangea · Sebastian Borgeaud · Charlie Nash · Mateusz Malinowski · Sander Dieleman · Oriol Vinyals · Matthew Botvinick · Ian Simon · Hannah Sheahan · Neil Zeghidour · Jean-Baptiste Alayrac · Joao Carreira · Jesse Engel

Thu Jul 21 08:40 AM -- 08:45 AM (PDT) @ Room 301 - 303
in DL: Generative Models and Autoencoders »

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64x64 ImageNet images and PG-19 books.

Keywords: Deep Learning · DL: Generative Models and Autoencoders · DL: Sequential Models, Time series

Author Information

Curtis Hawthorne (Google Brain)
Drew Jaegle (DeepMind)
Cătălina Cangea (DeepMind)
Sebastian Borgeaud (DeepMind)
Charlie Nash (DeepMind)
Mateusz Malinowski (DeepMind)
Sander Dieleman (DeepMind)
Oriol Vinyals (Google DeepMind)

Oriol Vinyals is a Research Scientist at Google. He works in deep learning with the Google Brain team. Oriol holds a Ph.D. in EECS from University of California, Berkeley, and a Masters degree from University of California, San Diego. He is a recipient of the 2011 Microsoft Research PhD Fellowship. He was an early adopter of the new deep learning wave at Berkeley, and in his thesis he focused on non-convex optimization and recurrent neural networks. At Google Brain he continues working on his areas of interest, which include artificial intelligence, with particular emphasis on machine learning, language, and vision.

Matthew Botvinick (DeepMind)
Ian Simon (Google Brain)
Hannah Sheahan (DeepMind)
Neil Zeghidour (Google)
Jean-Baptiste Alayrac (DeepMind)
Joao Carreira (DeepMind)
Jesse Engel (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors