Timezone: »
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it difficult to reuse old samples with large IS weights. In this paper, we consider PPO, a representative on-policy algorithm, and propose its improvement by dimension-wise IS weight clipping which separately clips the IS weight of each action dimension to avoid large bias and adaptively controls the IS weight to bound policy update from the current policy. This new technique enables efficient learning for high action-dimensional tasks and reusing of old samples like in off-policy learning to increase the sample efficiency. Numerical results show that the proposed new algorithm outperforms PPO and other RL algorithms in various Open AI Gym tasks.
Author Information
Seungyul Han (KAIST)
Youngchul Sung (KAIST)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Oral: Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning »
Wed. Jun 12th 06:35 -- 06:40 PM Room Hall B
More from the Same Authors
-
2022 : An Adaptive Entropy-Regularization Framework for Multi-Agent Reinforcement Learning »
WOOJUN KIM · Youngchul Sung -
2022 Poster: Robust Imitation Learning against Variations in Environment Dynamics »
Jongseong Chae · Seungyul Han · Whiyoung Jung · MYUNG-SIK CHO · Sungho Choi · Youngchul Sung -
2022 Spotlight: Robust Imitation Learning against Variations in Environment Dynamics »
Jongseong Chae · Seungyul Han · Whiyoung Jung · MYUNG-SIK CHO · Sungho Choi · Youngchul Sung -
2022 Poster: MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer »
JEON JEEWON · WOOJUN KIM · Whiyoung Jung · Youngchul Sung -
2022 Spotlight: MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer »
JEON JEEWON · WOOJUN KIM · Whiyoung Jung · Youngchul Sung -
2021 Poster: Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration »
Seungyul Han · Youngchul Sung -
2021 Spotlight: Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration »
Seungyul Han · Youngchul Sung