Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural gradient as vanilla policy gradients with a transformed advantage. RAT computes this transformation efficiently via randomized block Kaczmarz iterations on on-policy mini-batches, avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations. We provide convergence guarantees for RAT and demonstrate empirically that it matches or exceeds established natural-gradient methods across continuous and visual control benchmarks, while remaining simple to implement and compatible with various architectures.

Natural policy gradients are a powerful class of reinforcement learning methods that improve training stability by accounting for the geometry of the policy space. However, their adoption in modern deep reinforcement learning has been limited because they require estimating and inverting large Fisher matrices, which is computationally expensive and difficult to scale to large neural networks. This paper introduces Randomized Advantage Transformation (RAT), a new method that reformulates natural policy gradients into a standard policy gradient objective with a transformed advantage function. By leveraging the Woodbury formula and randomized iterative solvers, RAT avoids explicit Fisher matrix construction and enables natural-gradient-style optimization using only standard backpropagation. Experiments on continuous control and visual reinforcement learning benchmarks show that RAT achieves strong performance across diverse architectures, including shared actor–critic networks, while remaining simple, scalable, and computationally efficient.