Globally Convergent Offline Reinforcement Learning with Smoothed Bellman Residual Minimization

Bellman Residual Minimization (BRM) is a scalable gradient-based approach for offline reinforcement learning. While BRM is conceptually appealing in that it directly enforces Bellman consistency and admits stable optimization under neural network parametrization, existing BRM methods have not gained widespread adoption, largely because no prior work has established global convergence guarantees. In this work, we propose Off-GLADIUS, an offline reinforcement learning algorithm that provably converges to the global optimum. Our theoretical analysis builds on a reinterpretation of the recent loss-landscape results of Kang et al. [2025], which show that the BRM objective satisfies a Polyak–Łojasiewicz (PL) condition, thereby implying global optimality and linear convergence under gradient-based optimization. Empirically, in proof-of-concept experiments, our algorithm compares favorably with the prominent baselines Conservative Q-Learning (CQL) [Kumar et al., 2020] and OptiDICE [Lee et al., 2021], and outperforms the behavior policy on standard control benchmarks.