Timezone: »

 
Poster
PODS: Policy Optimization via Differentiable Simulation
Miguel Angel Zamora Mora · Momchil Peychev · Sehoon Ha · Martin Vechev · Stelian Coros

Tue Jul 20 09:00 AM -- 11:00 AM (PDT) @
in Poster Session 1 »

Current reinforcement learning (RL) methods use simulation models as simple black-box oracles. In this paper, with the goal of improving the performance exhibited by RL algorithms, we explore a systematic way of leveraging the additional information provided by an emerging class of differentiable simulators. Building on concepts established by Deterministic Policy Gradients (DPG) methods, the neural network policies learned with our approach represent deterministic actions. In a departure from standard methodologies, however, learning these policies does not hinge on approximations of the value function that must be learned concurrently in an actor-critic fashion. Instead, we exploit differentiable simulators to directly compute the analytic gradient of a policy's value function with respect to the actions it outputs. This, in turn, allows us to efficiently perform locally optimal policy improvement iterations. Compared against other state-of-the-art RL methods, we show that with minimal hyper-parameter tuning our approach consistently leads to better asymptotic behavior across a set of payload manipulation tasks that demand a high degree of accuracy and precision.

Keywords: Reinforcement Learning and Planning · Deep RL

Author Information

Miguel Angel Zamora Mora (ETH Zurich)
Momchil Peychev (ETH Zurich)
Sehoon Ha (Georgia Institute of Technology)
Martin Vechev (ETH Zurich)
Stelian Coros (ETH Zurich)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors