VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the value estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a \textit{general framework} that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. Our code is available at~\url{https://anonymous.4open.science/r/vipo2025-8FD4}.

Offline reinforcement learning enables an AI agent to learn optimal decision-making strategies exclusively from historical data, offering a safe alternative to costly real-world trial-and-error. AI agents are often trained using a "virtual simulator" built from this data, but because these simulators contain inherent errors, existing methods usually force the AI to be overly cautious and less effective. To address this, we introduce VIPO, a novel approach that improves simulator accuracy through a self-checking mechanism. VIPO works by comparing two estimates of a value function: one derived directly from the historical data and another predicted by the simulator itself. By minimizing the discrepancy between these two predictions during training, VIPO learns a significantly more reliable model. Our evaluations demonstrate that VIPO consistently outperforms existing methods across multiple complex benchmarks, providing a flexible framework that can be easily integrated to enhance the accuracy of current AI systems.