Reparameterization Proximal Policy Optimization

By leveraging differentiable dynamics, Reparameterization Policy Gradient (RPG) achieves high sample efficiency. However, current approaches are hindered by two critical limitations: the under-utilization of computationally expensive dynamics Jacobians and inherent training instability. While sample reuse offers a remedy for under-utilization, no prior principled framework exists, and naive attempts risks exacerbating instability. To address these challenges, we propose Reparameterization Proximal Policy Optimization (RPO). We first establish that under sample reuse, RPG naturally optimizes a PPO-style surrogate objective via Backpropagation Through Time, providing a unified framework for both on- and off-policy updates. To further ensure stability, RPO integrates a clipped policy gradient mechanism tailored for RPG and employs explicit Kullback-Leibler divergence regularization. Experimental results demonstrate that RPO maintains superior sample efficiency and consistently outperforms or achieves state-of-the-art performance across diverse tasks.

A family of policy gradient methods called reparameterization policy gradient (RPG) can learn efficiently by backpropagating through physics simulators. Yet today's RPG methods face two bottlenecks: they compute expensive gradients from the simulator but typically discard them after a single update. We propose Reparameterization Proximal Policy Optimization (RPO). RPO provides a principled way to reuse action-gradients multiple times without destabilizing training. Experiments show that RPO retains strong sample efficiency and consistently matches or exceeds state-of-the-art performance across diverse tasks. RPO helps bring efficient robot learning closer to practical deployment.