Harmonized Dual Policy Improvement for Modelic Reinforcement Learning

Policy-planner bootstrapping has emerged as a powerful paradigm in model-based reinforcement learning (MBRL). We formalize this process as a dual policy improvement mechanism synergizing: (i) exploitative improvement via off-policy $Q$-maximization, and (ii) lookahead improvement via planner alignment. While we theoretically prove that these improvements anchor to the same optimum, practical training process inevitably encounters gradient disagreement. Exacerbated by approximation inaccuracies and non-stationary data, this disagreement induces destructive interference in policy updates, destabilizing the bootstrapping loop and leading to suboptimal convergence. To address this, we propose harmonized dual policy improvement (HDPI), a gradient-level framework that reconciles exploitative and lookahead improvements through a harmonic optimization scheme. This scheme effectively maximizes the worst-case inner product between the harmonized update and the original gradients, ensuring directional consistency and stabilizing policy evolution. Extensive empirical evaluations on 14 challenging tasks from the DeepMind Control Suite and the Humanoid-Bench demonstrate that HDPI significantly enhances training stability and asymptotic performance, outperforming a wide range of strong baselines.

Modern AI systems that learn to make decisions, such as robots or autonomous agents, often rely on two complementary abilities: quickly reacting based on past experience, and planning ahead by imagining possible future outcomes. Recent research has shown that combining these two abilities can greatly improve performance. However, in practice, they often give conflicting learning signals, which can destabilize training and prevent the system from reaching its full potential. In this work, we study why this conflict happens and propose a new method to better coordinate the two learning processes. Our approach carefully balances short-term decision making with long-term planning so that they work together instead of interfering with each other. We also provide theoretical analysis showing that both learning strategies are ultimately trying to achieve the same goal. We evaluate our method on a range of challenging simulated control tasks, including humanoid and quadruped locomotion. The results show that our approach makes training more stable and significantly improves final performance compared with existing methods. These findings provide a more reliable way for future AI systems and robots to learn complex behaviors efficiently.