Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning

Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance. Code is available at https://github.com/Agentic-Intelligence-Lab/ROMBRL.

Many decision-making AI systems learn by trial and error, but in safety-critical settings such as robots or fusion reactors, it may be too risky or expensive to keep experimenting in the real world. Offline reinforcement learning instead learns from previously collected data, often by building a learned simulator, or “world model,” and training a policy inside it. A common problem is that this simulator is usually trained to copy the data, not to help the policy behave well when the real world is slightly different. As a result, policies that look strong in training can fail when sensors are noisy or the environment changes. We propose ROMBRL, a method that trains the policy and world model together: the policy tries to make good decisions, while the model is deliberately adjusted to expose difficult but plausible situations. This is like practicing with a coach who slightly changes the rules to reveal weaknesses before a real match. We also design efficient training tools so this back-and-forth process remains practical. In tests on robotic control benchmarks and simulated tokamak fusion control tasks, ROMBRL produced policies that kept higher performance under noise and other disturbances, suggesting a path toward more reliable learning from fixed datasets.