Timezone: »

 
Spotlight
On Proximal Policy Optimization's Heavy-tailed Gradients
Saurabh Garg · Joshua Zhanson · Emilio Parisotto · Adarsh Prasad · Zico Kolter · Zachary Lipton · Sivaraman Balakrishnan · Ruslan Salakhutdinov · Pradeep Ravikumar

Tue Jul 20 07:20 PM -- 07:25 PM (PDT) @
in Reinforcement Learning 8 »

Modern policy gradient algorithms such as Proximal Policy Optimization (PPO) rely on an arsenal of heuristics, including loss clipping and gradient clipping, to ensure successful learning. These heuristics are reminiscent of techniques from robust statistics, commonly used for estimation in outlier-rich ("heavy-tailed") regimes. In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function. We demonstrate that the gradients, especially for the actor network, exhibit pronounced heavy-tailedness and that it increases as the agent's policy diverges from the behavioral policy (i.e., as the agent goes further off policy). Further examination implicates the likelihood ratios and advantages in the surrogate reward as the main sources of the observed heavy-tailedness. We then highlight issues arising due to the heavy-tailed nature of the gradients. In this light, we study the effects of the standard PPO clipping heuristics, demonstrating that these tricks primarily serve to offset heavy-tailedness in gradients. Thus motivated, we propose incorporating GMOM, a high-dimensional robust estimator, into PPO as a substitute for three clipping tricks. Despite requiring less hyperparameter tuning, our method matches the performance of PPO (with all heuristics enabled) on a battery of MuJoCo continuous control tasks.

Keywords: Reinforcement Learning and Planning · Deep RL

Author Information

Saurabh Garg (Carnegie Mellon University)
Joshua Zhanson (Carnegie Mellon University)
Emilio Parisotto (Carnegie Mellon University)
Adarsh Prasad (Carnegie Mellon University)
Zico Kolter (Carnegie Mellon University / Bosch Center for AI)
Zachary Lipton (Carnegie Mellon University)
Sivaraman Balakrishnan (CMU)
Ruslan Salakhutdinov (Carnegie Mellen University)
Pradeep Ravikumar (Carnegie Mellon University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors