Skip to yearly menu bar Skip to main content


Poster

Collaborative Evolutionary Reinforcement Learning

Shauharda Khadka · Somdeb Majumdar · Tarek Nassar · Zach Dwiel · Evren Tumer · Santiago Miret · Yinyin Liu · Kagan Tumer

Pacific Ballroom #45

Keywords: [ Deep Reinforcement Learning ] [ Ensemble Methods ] [ Parallel and Distributed Learning ] [ Planning and Control ] [ Theory and Algorithms ]


Abstract:

Deep reinforcement learning algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically struggle with achieving effective exploration and are extremely sensitive to the choice of hyperparameters. One reason is that most approaches use a noisy version of their operating policy to explore - thereby limiting the range of exploration. In this paper, we introduce Collaborative Evolutionary Reinforcement Learning (CERL), a scalable framework that comprises a portfolio of policies that simultaneously explore and exploit diverse regions of the solution space. A collection of learners - typically proven algorithms like TD3 - optimize over varying time-horizons leading to this diverse portfolio. All learners contribute to and use a shared replay buffer to achieve greater sample efficiency. Computational resources are dynamically distributed to favor the best learners as a form of online algorithm selection. Neuroevolution binds this entire process to generate a single emergent learner that exceeds the capabilities of any individual learner. Experiments in a range of continuous control benchmarks demonstrate that the emergent learner significantly outperforms its composite learners while remaining overall more sample-efficient - notably solving the Mujoco Humanoid benchmark where all of its composite learners (TD3) fail entirely in isolation.

Live content is unavailable. Log in and register to view live content