Timezone: »
Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that more information is needed to accurately estimate the value function than to learn the optimal policy. Consequently, the use of a shared representation for the policy and value function can lead to overfitting. To alleviate this problem, we propose two approaches which are combined to create IDAAC: Invariant Decoupled Advantage Actor-Critic. First, IDAAC decouples the optimization of the policy and value function, using separate networks to model them. Second, it introduces an auxiliary loss which encourages the representation to be invariant to task-irrelevant properties of the environment. IDAAC shows good generalization to unseen environments, achieving a new state-of-the-art on the Procgen benchmark and outperforming popular methods on DeepMind Control tasks with distractors. Our implementation is available at https://github.com/rraileanu/idaac.
Author Information
Roberta Raileanu (NYU)
Rob Fergus (Facebook / NYU)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: Decoupling Value and Policy for Generalization in Reinforcement Learning »
Tue. Jul 20th 04:00 -- 06:00 PM Room Virtual
More from the Same Authors
-
2023 : Accelerating exploration and representation learning with offline pre-training »
Bogdan Mazoure · Jake Bruce · Doina Precup · Rob Fergus · Ankit Anand -
2023 Poster: Distilling Internet-Scale Vision-Language Models into Embodied Agents »
Theodore R Sumers · Kenneth Marino · Arun Ahuja · Rob Fergus · Ishita Dasgupta -
2023 Poster: Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC »
Yilun Du · Conor Durkan · Robin Strudel · Josh Tenenbaum · Sander Dieleman · Rob Fergus · Jascha Sohl-Dickstein · Arnaud Doucet · Will Grathwohl -
2021 Poster: Reinforcement Learning with Prototypical Representations »
Denis Yarats · Rob Fergus · Alessandro Lazaric · Lerrel Pinto -
2021 Spotlight: Reinforcement Learning with Prototypical Representations »
Denis Yarats · Rob Fergus · Alessandro Lazaric · Lerrel Pinto -
2020 : Automatic Data Augmentation for Generalization in Reinforcement Learning »
Roberta Raileanu -
2020 Poster: Fast Adaptation to New Environments via Policy-Dynamics Value Functions »
Roberta Raileanu · Max Goldstein · Arthur Szlam · Facebook Rob Fergus -
2018 Poster: Stochastic Video Generation with a Learned Prior »
Emily Denton · Rob Fergus -
2018 Oral: Stochastic Video Generation with a Learned Prior »
Emily Denton · Rob Fergus -
2018 Poster: Modeling Others using Oneself in Multi-Agent Reinforcement Learning »
Roberta Raileanu · Emily Denton · Arthur Szlam · Facebook Rob Fergus -
2018 Oral: Modeling Others using Oneself in Multi-Agent Reinforcement Learning »
Roberta Raileanu · Emily Denton · Arthur Szlam · Facebook Rob Fergus