How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation

Augustine Mavor-Parker · Kimberly Young · Caswell Barry · Lewis Griffin

Hall E #837

Keywords: [ APP: Neuroscience, Cognitive Science ] [ RL: Deep RL ]

[ Abstract ]
[ Paper PDF
Wed 20 Jul 3:30 p.m. PDT — 5:30 p.m. PDT
Spotlight presentation: Reinforcement Learning: Deep RL
Wed 20 Jul 7:30 a.m. PDT — 9 a.m. PDT


When extrinsic rewards are sparse, artificial agents struggle to explore an environment. Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but it is known to fail when faced with action-dependent noise sources (‘noisy TVs’). In an attempt to make exploring agents robust to Noisy TVs, we present a simple solution: aleatoric mapping agents (AMAs). AMAs are a novel form of curiosity that explicitly ascertain which state transitions of the environment are unpredictable, even if those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states, with the aim of reducing intrinsic rewards for those transitions that are unpredictable. We demonstrate that in a range of environments AMAs are able to circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents. Furthermore, we demonstrate empirically that other common exploration approaches---previously thought to be immune to agent-induced randomness---can be trapped by stochastic dynamics.

Chat is not available.