AdamO: A Collapse-Suppressed Optimizer for Offline RL

Offline reinforcement learning (RL) can fail spectacularly when bootstrapped temporal-difference (TD) updates amplify their own errors, driving the critic toward extreme and unusable Q-values. A key counterintuitive insight of this work is that collapse is not only a property of the backup rule or network architecture: the optimizer alone can directly trigger or suppress instability. From a control-theoretic viewpoint, we model offline TD learning as a feedback system and analyze Adam-based critic updates. This yields a necessary and sufficient condition for stability: training is stable if and only if the spectral radius of the update dynamics is strictly below one. Further analysis reveals that standard Adam updates can inadvertently distort the parameter geometry, necessitating explicit orthogonality constraints to prevent TD error amplification. To this end, we propose AdamO, an Adam-based optimizer with a decoupled orthogonality correction regulated by a strict task-alignment budget. We prove that this design theoretically guarantees worst-case task safety and preserves Adam’s continuous-time dissipative dynamics. Empirically, AdamO is plug-and-play across diverse offline RL baselines, improving stability and returns across a broad suite of benchmarks.

Offline reinforcement learning learns from logged data instead of interacting with the environment, but it can become unstable when its own prediction errors keep feeding back into training. Our paper shows that this instability can come not only from the learning algorithm itself, but also from the optimizer used to update the model. We introduce AdamO, a modified version of Adam that adds a controlled orthogonality correction to reduce these harmful feedback loops. Across a range of benchmark tasks, AdamO makes training more stable and often improves the final performance of offline reinforcement learning methods.