Population-Free Pareto Tracking for Sample-Efficient Multi-Policy MORL

Multi-objective reinforcement learning (MORL) is a fundamental framework for real-world decision-making problems involving multiple conflicting criteria. Existing multi-policy (MP) methods typically rely on online evolutionary frameworks that maintain large policy populations, leading to high sample complexity and excessive agent–environment interactions. To mitigate these limitations, we present Multi-policy Pareto Front Tracking (MPFT), a framework without a self-evolving population. It leverages an efficient Pareto-tracking mechanism initialized with single-objective extreme policies to trace the Pareto front, and further densifies sparse regions to achieve an accurate approximation of the full Pareto front. MPFT can be seamlessly integrated with advanced offline MORL algorithms, thereby substantially improving sample efficiency. We evaluate MPFT on six robotic control tasks with up to three objectives and three high-dimensional tasks with more than three objectives. Experimental results show that MPFT outperforms state-ofthe-art baselines in terms of hypervolume and expected utility. It also significantly reduces agent–environment interactions. These results further demonstrate that MPFT serves as a general purpose framework that can seamlessly integrate both online and offline MORL algorithms.

In real-world applications, decision-making often involves balancing conflicting goals. For example, programming a robot to run faster inherently drains its battery more quickly. To address this, we aim to learn a comprehensive “menu” of optimal trade-offs—ranging from maximum speed to maximize energy efficiency—so that users can freely choose the option that best suits their current needs. However, existing methods discover this menu by maintaining huge, self-evolving pools of possible solutions (we call it "populations"). This process requires an impractical amount of trial-and-error and data, making real-world deployment extremely expensive and difficult. Our paper introduces a much faster approach that eliminates the need for these massive populations. Instead, our method acts like an explorer methodically tracing the edge of a map. It first finds the absolute extremes (e.g., the single fastest policy and the most efficient policy), and then intelligently "tracks" the boundary connecting them to uncover all the best compromises in between. This drastically reduces the data and interactions needed to train the policy, making it easier to deploy in the real world.