Timezone: »

 
Oral
Quantile Credit Assignment
Thomas Mesnard · Wenqi Chen · Alaa Saade · Yunhao Tang · Mark Rowland · Theophane Weber · Clare Lyle · Audrunas Gruslys · Michal Valko · Will Dabney · Georg Ostrovski · Eric Moulines · Remi Munos

Tue Jul 25 09:10 PM -- 09:18 PM (PDT) @ Ballroom C
in Oral A6 Reinforcement Learning 2 »

In reinforcement learning, the credit assignment problem is to distinguish luck from skill, that is, separate the inherent randomness in the environment from the controllable effects of the agent's actions. This paper proposes two novel algorithms, Quantile Credit Assignment (QCA) and Hindsight QCA (HQCA), which incorporate distributional value estimation to perform credit assignment. QCA uses a network that predicts the quantiles of the return distribution, whereas HQCA additionally incorporates information about the future. Both QCA and HQCA have the appealing interpretation of leveraging an estimate of the quantile level of the return (interpreted as the level of "luck") in order to derive a "luck-dependent" baseline for policy gradient methods. We show theoretically that this approach gives an unbiased policy gradient estimate that can yield significant variance reductions over a standard value estimate baseline. QCA and HQCA significantly outperform prior state-of-the-art methods on a range of extremely difficult credit assignment problems.

[ PDF] 

Author Information

Thomas Mesnard (Google DeepMind)
Wenqi Chen (Harvard University)
Alaa Saade (DeepMind)
Yunhao Tang (Google DeepMind)
Mark Rowland (Google DeepMind)
Theophane Weber (Google DeepMind)
Clare Lyle (University of Oxford)
Audrunas Gruslys (DeepMind)
Michal Valko (Google DeepMind / Inria / MVA)
Will Dabney (Google DeepMind)
Georg Ostrovski (DeepMind)
Eric Moulines (Ecole Polytechnique)
Remi Munos (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors