Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment. Current approaches attempt to enforce smoothness by directly regularizing the policy's output. We argue that this approach treats the symptom rather than the cause. In this work, we theoretically establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. By applying implicit differentiation to the actor-critic objective, we prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). To empirically validate this theoretical insight, we introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework that treats the critic as a scalar field and stabilizes its induced action-gradient field. PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature. Experimental results demonstrate that PAVE achieves smoothness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance, without modifying the actor.

When AI agents learn to control physical systems---robots, drones, or autonomous vehicles---they often produce jerky, oscillating actions. These erratic movements waste energy, accelerate hardware wear, and can be unsafe. Current approaches try to smooth the agent's output directly, but this only treats the symptom: the underlying learning signal still points in conflicting directions for nearly identical situations, forcing the agent into a tug-of-war between performing well and behaving smoothly. We identified the true source of this jerkiness: the unstable geometric shape of the value function that the agent uses to evaluate its options. Using mathematical analysis, we proved that smooth behavior emerges naturally when this internal landscape is stable rather than jagged. We then built PAVE, a method that gently sculpts the agent's evaluation landscape during training, without modifying the decision-making mechanism itself. Across six standard robotics benchmarks, PAVE produced smoother and more consistent control than prior methods while maintaining or improving task performance---a step toward safer deployment of learned controllers on real hardware.