Timezone: »

 
Poster
The Primacy Bias in Deep Reinforcement Learning
Evgenii Nikishin · Max Schwarzer · Pierluca D'Oro · Pierre-Luc Bacon · Aaron Courville

Tue Jul 19 03:30 PM -- 05:30 PM (PDT) @ Hall E #829
in Poster Session 1 »

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

Keywords: Reinforcement Learning · RL: Deep RL

Author Information

Evgenii Nikishin (Mila, Université de Montréal)
Max Schwarzer (Mila, Google Brain)
Pierluca D'Oro (Mila, Université de Montréal)
Pierre-Luc Bacon (Mila)
Aaron Courville (Université de Montréal)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors