CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

In open-ended and changing environments, agents face a wide range of potential tasks that might not come with associated reward functions. Such autonomous learning agents must set their own tasks and build their own curriculum through an intrinsically motivated exploration. Because some tasks might prove easy and some impossible, agents must actively select which task to practice at any given moment to maximize their overall mastery on the set of learnable tasks. This paper proposes CURIOUS, an algorithm that leverages: 1) an extension of Universal Value Function Approximators to achieve within a unique policy, multiple tasks, each parameterized by multiple goals and 2) an automated curriculum learning mechanism that biases the attention of the agent towards tasks maximizing the absolute learning progress. Agents focus on achievable tasks first, and focus back on tasks that are being forgotten. Experiments conducted in a new multi-task multi-goal robotic environment show that our algorithm benefits from these two ideas and demonstrate properties of robustness to distracting tasks, forgetting and changes in body properties.