Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning (MORL) seeks policies that effectively balance conflicting objectives. However, presenting many diverse policies without accounting for the decision maker’s (DM’s) preferences can overwhelm the decision-making process. On the other hand, accurately specifying preferences in advance is often unrealistic. To address these challenges, we introduce a human-in-the-loop MORL framework that interactively discovers preferred policies during optimization. Our approach proactively learns the DM’s implicit preferences in real time, requiring no a priori knowledge. Importantly, we integrate this preference learning directly into a parallel optimization framework, balancing exploration and exploitation to identify high-quality policies aligned with the DM's preferences. Evaluations on a complex quadrupedal robot simulation environment demonstrate that, with only interactions, our proposed method can identify policies aligned with human preferences, e.g., running like a dog. Further experiments on seven MuJoCo tasks and a multi-microgrid system design task against eight state-of-the-art MORL algorithms fully demonstrate the effectiveness of our proposed framework. Demonstrations and full experiments are in https://sites.google.com/view/pbmorl/home.

Many real-world decision-making problems involve balancing conflicting goals. For example, a robot may need to move quickly while also conserving energy, or an energy system may need to balance cost, stability, and efficiency. Existing AI methods usually require humans to specify these trade-offs in advance using numerical reward weights, but in practice people often do not know exactly what they want until they see different behaviors. We developed a human-in-the-loop reinforcement learning framework called PBMORL that learns a person’s preferences interactively during training. Instead of asking users to design complicated reward functions, our method simply asks them to compare pairs of candidate behaviors and indicate which one they prefer. PBMORL then learns these preferences and uses them to guide the search toward more desirable solutions while still exploring alternative possibilities. We tested PBMORL on challenging robot control tasks and energy system design problems. In experiments with a quadruped robot, the method discovered more natural and energy-efficient running behaviors that better matched human preferences than existing approaches. Our work provides a practical way to align AI decision-making systems with human intentions in complex multi-objective problems.