Timezone: »

 
Poster
Resurrecting Recurrent Neural Networks for Long Sequences
Antonio Orvieto · Samuel Smith · Albert Gu · Anushan Fernando · Caglar Gulcehre · Razvan Pascanu · Soham De

Wed Jul 26 02:00 PM -- 03:30 PM (PDT) @ Exhibit Hall 1 #521
in Poster Session 3 »

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. We show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring careful normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, and introduce an RNN block called the Linear Recurrent Unit (or LRU) that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

[ Poster] 

Author Information

Antonio Orvieto (ETH Zurich)
Samuel Smith (Google DeepMind)
Albert Gu (Carnegie Mellon University, DeepMind)
Anushan Fernando (DeepMind)
Caglar Gulcehre (Google DeepMind)
Razvan Pascanu (DeepMind)
Soham De (Google DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors