Timezone: »
It has been a recent trend to leverage the power of supervised learning (SL) towards more effective reinforcement learning (RL) methods. We propose a novel phasic solution by alternating online RL and offline SL for tackling sparse-reward goal-conditioned problems. In the online phase, we perform RL training and collect rollout data while in the offline phase, we perform SL on those successful trajectories from the dataset. To further improve sample efficiency, we adopt additional techniques in the online phase including task reduction to generate more feasible trajectories and a value-difference-based intrinsic reward to alleviate the sparse-reward issue. We call this overall framework, PhAsic self-Imitative Reduction (PAIR). PAIR is compatible with various online and offline RL methods and substantially outperforms both non-phasic RL and phasic SL baselines on sparse-reward robotic control problems, including a particularly challenging stacking task. PAIR is the first RL method that learns to stack 6 cubes with only 0/1 success rewards from scratch.
Author Information
Yunfei Li (Tsinghua University)
Tian Gao (Tsinghua University)
Jiaqi Yang (University of California, Berkeley)
Huazhe Xu (Stanford University)
Yi Wu (UC Berkeley)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning »
Wed. Jul 20th through Thu the 21st Room Hall E #820
More from the Same Authors
-
2021 : Disentangled Attention as Intrinsic Regularization for Bimanual Multi-Object Manipulation »
Minghao Zhang · Pingcheng Jian · Yi Wu · Harry (Huazhe) Xu · Xiaolong Wang -
2022 : Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning »
Zhecheng Yuan · Zhecheng Yuan · Zhengrong Xue · Zhengrong Xue · Bo Yuan · Bo Yuan · Xueqian Wang · Xueqian Wang · Yi Wu · Yi Wu · Yang Gao · Yang Gao · Huazhe Xu · Huazhe Xu -
2023 : 🎤 SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores »
Zhiyu Mei · Wei Fu · Guangju Wang · Huanchen Zhang · Yi Wu -
2022 Poster: Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification »
Ling Pan · Longbo Huang · Tengyu Ma · Huazhe Xu -
2022 Spotlight: Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification »
Ling Pan · Longbo Huang · Tengyu Ma · Huazhe Xu -
2022 Poster: Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning »
Wei Fu · Chao Yu · Zelai Xu · Jiaqi Yang · Yi Wu -
2022 Spotlight: Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning »
Wei Fu · Chao Yu · Zelai Xu · Jiaqi Yang · Yi Wu -
2018 Poster: Discrete-Continuous Mixtures in Probabilistic Programming: Generalized Semantics and Inference Algorithms »
Yi Wu · Siddharth Srivastava · Nicholas Hay · Simon Du · Stuart Russell -
2018 Oral: Discrete-Continuous Mixtures in Probabilistic Programming: Generalized Semantics and Inference Algorithms »
Yi Wu · Siddharth Srivastava · Nicholas Hay · Simon Du · Stuart Russell