Skip to yearly menu bar Skip to main content


Hessian Aided Policy Gradient

Zebang Shen · Alejandro Ribeiro · Hamed Hassani · Hui Qian · Chao Mi

Pacific Ballroom #114

Keywords: [ Theory and Algorithms ] [ Non-convex Optimization ]

Abstract: Reducing the variance of estimators for policy gradient has long been the focus of reinforcement learning research. While classic algorithms like REINFORCE find an $\epsilon$-approximate first-order stationary point in $\OM({1}/{\epsilon^4})$ random trajectory simulations, no provable improvement on the complexity has been made so far. This paper presents a Hessian aided policy gradient method with the first improved sample complexity of $\OM({1}/{\epsilon^3})$. While our method exploits information from the policy Hessian, it can be implemented in linear time with respect to the parameter dimension and is hence applicable to sophisticated DNN parameterization. Simulations on standard tasks validate the efficiency of our method.

Live content is unavailable. Log in and register to view live content