Timezone: »

 
Oral
Reparameterized Policy Learning for Multimodal Trajectory Optimization
Zhiao Huang · Litian Liang · Zhan Ling · Xuanlin Li · Chuang Gan · Hao Su

Tue Jul 25 08:46 PM -- 08:54 PM (PDT) @ Ballroom B
in Oral A5 Reinforcement Learning 1 »

We investigate the challenge of parametrizing policies for reinforcement learning (RL) in high-dimensional continuous action spaces. Our objective is to develop a multimodal policy that overcomes limitations inherent in the commonly-used Gaussian parameterization. To achieve this, we propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. By conditioning the policy on a latent variable, we derive a novel variational bound as the optimization objective, which promotes exploration of the environment. We then present a practical model-based RL method, called Reparameterized Policy Gradient (RPG), which leverages the multimodal policy parameterization and learned world model to achieve strong exploration capabilities and high data efficiency. Empirical results demonstrate that our method can help agents evade local optima in tasks with dense rewards and solve challenging sparse-reward environments by incorporating an object-centric intrinsic reward. Our method consistently outperforms previous approaches across a range of tasks. Code and supplementary materials are available on the project page https://haosulab.github.io/RPG/

[ PDF] 

Author Information

Zhiao Huang (UCSD)
Litian Liang (University of California, San Diego)
Zhan Ling (UC San Diego)
Xuanlin Li (UCSD)
Chuang Gan (Umass Amherst/ IBM)
Hao Su (UCSD)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors