Timezone: »

 
Spotlight
Improving Policy Optimization with Generalist-Specialist Learning
Zhiwei Jia · Xuanlin Li · Zhan Ling · Shuang Liu · Yiran Wu · Hao Su

Wed Jul 20 08:55 AM -- 09:00 AM (PDT) @ Room 307
in Reinforcement Learning: Deep RL »

Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations. We empirically observe that an agent trained on many variations (a generalist) tends to learn faster at the beginning, yet its performance plateaus at a less optimal level for a long time. In contrast, an agent trained only on a few variations (a specialist) can often achieve high returns under a limited computational budget. To have the best of both worlds, we propose a novel generalist-specialist training framework. Specifically, we first train a generalist on all environment variations; when it fails to improve, we launch a large population of specialists with weights cloned from the generalist, each trained to master a selected small subset of variations. We finally resume the training of the generalist with auxiliary rewards induced by demonstrations of all specialists. In particular, we investigate the timing to start specialist training and compare strategies to learn generalists with assistance from specialists.We show that this framework pushes the envelope of policy learning on several challenging and popular benchmarks including Procgen, Meta-World and ManiSkill.

Keywords: RL: Deep RL

Author Information

Zhiwei Jia (University of California, San Diego)
Xuanlin Li (UCSD)
Zhan Ling (UCSD)
Shuang Liu (University of California, San Diego)
Yiran Wu (University of California, San Diego)
Hao Su (UCSD)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors