The Surprising Difficulty of Search in Model-Based Reinforcement Learning

This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains. Code can be found at https://github.com/facebookresearch/MRSQ.

Model-based reinforcement learning promises to let AI agents learn efficiently by simulating the consequences of their actions, yet it has long underperformed simpler approaches. The conventional explanation blames inaccurate learned models, arguing that small prediction errors compound over time, making long-term predictions inaccurate. We challenge this view, demonstrating that search can actually harm performance even when the model is highly accurate. The real culprit is overestimation bias, where value estimates that guide planning become inflated when search explores actions outside the training distribution. To address this, we use an ensemble of value functions and take their minimum, producing cautiously pessimistic estimates that keep search grounded. We instantiate this insight in a new algorithm called MRS.Q and evaluate it across over 50 tasks spanning multiple benchmark domains. With a single fixed set of hyperparameters, MRS.Q achieves state-of-the-art results, outperforming both leading model-free and model-based methods. Our findings reframe the research agenda, showing that the bottleneck isn't building better models, but rather understanding how to use them effectively during planning.