Timezone: »

 
Oral
Addressing Function Approximation Error in Actor-Critic Methods
Scott Fujimoto · Herke van Hoof · David Meger

Thu Jul 12 06:20 AM -- 06:30 AM (PDT) @ A1
in Reinforcement Learning 10 »

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

[ PDF]  [ Video] 

Author Information

Scott Fujimoto (McGill University)
Herke van Hoof (McGill University)
David Meger (McGill University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors