Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization. We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously. We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds. MOPO operates directly on pairwise preferences without point-wise rewards, and admits simple closed-form iterative updates. Empirically, MOPO recovers Pareto-optimal policies on synthetic benchmarks and, when fine-tuned on human-preference data, yields multi-billion parameter models that achieve higher rewards and Pareto-dominate baselines, with stable and robust optimization dynamics.

When we train large language models (like chatbots), we usually try to teach them to follow human preferences using a single score. However, real-world human values often conflict. For example, we want an AI to be as helpful as possible, but we also need it to be completely harmless. Boosting one of these qualities can accidentally cause the other to drop, and there is no single "perfect balance" that fits every situation. To solve this, we introduced Multi-Objective Preference Optimization (MOPO). Instead of forcing different goals into a single score, MOPO lets trainers pick a main goal (like helpfulness) while setting strict minimum boundaries for other goals (like safety thresholds). Our method works directly with simple feedback on which answers humans prefer, skipping the complicated step of building a separate scoring system. In our tests, MOPO successfully balanced these conflicting goals, creating highly stable, multi-billion-parameter models that outperform existing methods without accidentally compromising on safety.