Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator

Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses. We study this phenomenon through the noise-to-signal ratio (NSR) of a policy-gradient estimator, defined as the estimator variance (noise) normalized by the squared norm of the true gradient (signal). Our main result is that, for (i) finite-horizon linear systems with Gaussian policies and linear state-feedback, and (ii) finite-horizon polynomial systems with Gaussian policies and polynomial feedback, the NSR of the REINFORCE estimator can be characterized exactly—either in closed form or via numerical moment-evaluation algorithms—without approximation. For general nonlinear dynamics and expressive policies (including neural policies), we further derive a general upper bound on the variance. These characterizations enable a direct examination of how NSR varies across policy parameters and how it evolves along optimization trajectories (e.g. SGD and Adam). Across a range of examples, we find that the NSR landscape is highly non-uniform and typically increases as the policy approaches an optimum; in some regimes it blows up, which can trigger training instability and policy collapse.

Reinforcement learning with policy-gradient methods can become unreliable when a system gets close to a good solution. The issue is that each training step relies on a noisy estimate of how to improve the policy, and this noise can overwhelm the useful learning signal. We study this effect through the “noise-to-signal ratio,” which measures how noisy a policy-gradient estimate is relative to the true direction of improvement. For several important control settings, including linear and polynomial systems, we show how to compute this ratio exactly rather than relying on expensive estimation. For more general nonlinear systems, including those with neural-network policies, we derive bounds that reveal what makes the noise large. Our results show that the noise-to-signal ratio is non-uniform: it can grow sharply as the policy improves, especially when the policy becomes nearly deterministic or when the system dynamics are unstable. This helps explain why reinforcement-learning algorithms may slow down, oscillate, or collapse near good solutions, and it points toward more reliable ways to control exploration and reduce gradient noise.