REG: In-Sample RL via Regularizing the Evaluation Gap

Distribution shift poses a fundamental challenge in offline reinforcement learning, often leading to value overestimation when querying out-of-distribution actions. We introduce Regularized Evaluation Gap (REG) as a bridge between implicit methods like IQL and explicit conservative methods. We formulate policy evaluation as a robust optimization problem over an ambiguity set of critics and show that IQL’s objective can be viewed as an approximate dual solution to this problem. To extract a policy from the learned value function, we propose a practical Orthogonal Policy Gradient (OPG) update. This method regularizes an aggressive, mode-seeking policy gradient by projecting it onto the subspace orthogonal to a stable, in-sample behavior cloning gradient. Extensive D4RL experiments demonstrate that REG matches state-of-the-art performance among both Gaussian methods and diffusion-based approaches without the computational burden of the latter.

Reinforcement learning trains computer programs to make sequences of decisions, such as steering a self-driving car or recommending a treatment plan. In many real-world settings, letting the program experiment freely is too costly or dangerous, so it must learn entirely from a fixed log of past decisions and their outcomes. The fundamental difficulty is that the program will inevitably consider actions nobody recorded — and when it tries to predict how good those untried actions would be, it becomes wildly overconfident, which leads to poor performance after deployment. We developed a method called Regularized Evaluation Gap (REG) that turns this difficulty into a precise mathematical problem: find a strategy whose performance can be reliably estimated even when we are uncertain about untried actions. Solving it yields a simple "one-sided" learning rule. We pair this with a new update that gently nudges the program toward better choices without drifting beyond what the data actually supports. On standard benchmarks, REG matches or beats much heavier modern approaches while remaining simple and fast, making safer offline learning more practical for real-world applications like robotics and healthcare.