Timezone: »

 
Exploration via Empowerment Gain: Combining Novelty, Surprise and Learning Progress
Philip Becker-Ehmck · Maximilian Karl · Jan Peters · Patrick van der Smagt

@
in ICML 2021 Workshop on Unsupervised Reinforcement Learning »

Exploration in the absence of a concrete task is a key characteristic of autonomous agents and vital for the emergence of intelligent behaviour. Various intrinsic motivation frameworks have been suggested, such as novelty seeking, surprise maximisation or empowerment. Here we focus on the latter, empowerment, an agent-centric and information-theoretic measure of an agent's perceived influence on the world. By considering improving one's empowerment estimator - we call it empowerment gain (EG) - we derive a novel exploration criterion that focuses directly on the desired goal: exploration in order to help the agent recognise its capability to interact with the world. We propose a new theoretical framework based on improving a parametrised estimation of empowerment and show how it integrates novelty, surprise and learning progress into a single formulation. Empirically, we validate our theoretical findings on some simple but instructive grid world environments. We show that while such an agent is still novelty seeking, i.e. interested in exploring the whole state space, it focuses on exploration where its perceived influence is greater, avoiding areas of greater stochasticity or traps that limit its control.

Author Information

Philip Becker-Ehmck (Machine Learning Research Lab, Volkswagen Group)
Maximilian Karl (Data Lab, Volkswagen Group)
Jan Peters (TU Darmstadt)
Patrick van der Smagt (Volkswagen Group)

More from the Same Authors