Timezone: »

 
Poster
Understanding Self-Predictive Learning for Reinforcement Learning
Yunhao Tang · Zhaohan Guo · Pierre Richemond · Bernardo Avila Pires · Yash Chandak · Remi Munos · Mark Rowland · Mohammad Gheshlaghi Azar · Charline Le Lan · Clare Lyle · Andras Gyorgy · Shantanu Thakoor · Will Dabney · Bilal Piot · Daniele Calandriello · Michal Valko

Thu Jul 27 04:30 PM -- 06:00 PM (PDT) @ Exhibit Hall 1 #308
in Poster Session 6 »

We study the learning dynamics of self-predictive learning for reinforcement learning, a family of algorithms that learn representations by minimizing the prediction error of their own future latent representations. Despite its recent empirical success, such algorithms have an apparent defect: trivial representations (such as constants) minimize the prediction error, yet it is obviously undesirable to converge to such solutions. Our central insight is that careful designs of the optimization dynamics are critical to learning meaningful representations. We identify that a faster paced optimization of the predictor and semi-gradient updates on the representation, are crucial to preventing the representation collapse. Then in an idealized setup, we show self-predictive learning dynamics carries out spectral decomposition on the state transition matrix, effectively capturing information of the transition dynamics. Building on the theoretical insights, we propose bidirectional self-predictive learning, a novel self-predictive algorithm that learns two representations simultaneously. We examine the robustness of our theoretical insights with a number of small-scale experiments and showcase the promise of the novel representation learning algorithm with large-scale experiments.

[ PDF] 

Author Information

Yunhao Tang (Google DeepMind)
Zhaohan Guo (DeepMind)
Pierre Richemond (Google DeepMind)
Bernardo Avila Pires (Google DeepMind)
Yash Chandak (Stanford University)
Remi Munos (DeepMind)
Mark Rowland (Google DeepMind)
Mohammad Gheshlaghi Azar (Google DeepMind)
Charline Le Lan (University of Oxford)
Clare Lyle (University of Oxford)
Andras Gyorgy (Google DeepMind)
Shantanu Thakoor (DeepMind)
Will Dabney (Google DeepMind)
Bilal Piot (Google DeepMind)
Daniele Calandriello (DeepMind)
Michal Valko (Google DeepMind / Inria / MVA)

More from the Same Authors