Doubly Regularized Markov Decision Processes for Robust Reinforcement Learning

Empirical successes show that regularization improves the stability and efficiency of reinforcement learning (RL), with applications in robotics and post-training of large language models. Yet, theoretical analyses of regularized Markov decision processes (MDPs) have mostly been confined to the standard RL setting. In this work, we investigate regularized MDPs through the lens of robust RL. We introduce a doubly regularized MDP framework that combines policy and dynamics regularizations, enabling robust policy learning while naturally accommodating continuous action spaces. Within this framework, we develop an optimism-based online algorithm and provide the first finite-sample regret guarantees in both tabular and linear settings. Our results show that algorithms for doubly regularized MDPs are as sample-efficient as well-studied robust MDP algorithms, while additionally benefiting from the flexibility of soft policies. We further design practical algorithmic variants for both settings and demonstrate empirically that our approach efficiently and effectively handles function approximation and exploration in large state-action spaces, achieving robust performances.

We propose a new reinforcement learning framework that makes agents more robust to uncertainty. Our method combines two types of regularization, which can be viewed as soft constraints on the learning process. Policy regularization improves reward robustness, enabling agents to remain effective even when rewards are noisy or misspecified, while transition regularization improves transition robustness by accounting for possible shifts or errors in the environment dynamics. Building on this framework, we develop a new algorithm, Robust Soft Policy Value Iteration (RSPVI), which learns soft stochastic policies instead of the deterministic policies commonly used in prior work. This makes the approach naturally applicable to continuous action spaces while also improving training stability and maintaining sample efficiency. Experiments on robotics-style control tasks show that our method performs better than standard reinforcement learning approaches when the environment is noisy or perturbed, suggesting its potential for building more reliable real-world systems.