Taming Aleatoric Impulse in Off-Policy Reinforcement Learning

Off-policy reinforcement learning is vulnerable to overestimation bias, which is rooted in the total value uncertainty. However, existing methods typically misaddress this by targeting the epistemic component, neglecting the aleatoric component. We identify for the first time that this oversight fails to contain a massive bias surge, termed Aleatoric Impulse. Although transient, this impulse fundamentally derails the learning trajectory, permanently locking the agent into suboptimal policies. To counteract this, we propose Aleatoric Impulse Damping (AID), the first mechanism that models total value uncertainty by disentangling the return variance into epistemic and aleatoric components, followed by their adaptive weighted recombination. Leveraging this derived uncertainty, the critic constructs a pessimistic lower confidence bound to surgically suppress the impulse. Complementing this, the actor utilizes a symmetrical upper confidence bound to drive optimistic exploration, ensuring that the necessary pessimism does not compromise exploration efficiency. We integrate this mechanism into the Distributional Soft Actor-Critic algorithm to establish DSAC-AID. Extensive experiments on the high-dimensional Gym-MuJoCo and DeepMind Control Suite benchmarks demonstrate that it achieves state-of-the-art results in final performance.

To master complex systems like robots or autonomous vehicles, artificial intelligence is often trained using a trial-and-error approach. However, we discovered a major flaw in this process. During early training, the AI frequently mistakes random chance—essentially pure luck—for actual skill. We identified this phenomenon as "aleatoric impulse". This sudden surge of false overconfidence permanently derails the AI, causing it to lock into poor, unsafe behaviors. To solve this, we designed a self-correcting mechanism that helps the AI explicitly distinguish between its own lack of knowledge and purely unpredictable randomness. By accurately quantifying these uncertainties, our method forces the AI to be cautious about lucky flukes while remaining curious enough to explore new, better solutions. We tested this on highly complex simulated robots and found it significantly outperforms existing methods. Ultimately, our research allows us to train much more stable and reliable AI systems, ensuring they can safely navigate the chaotic, unpredictable nature of the real world.