Pareto-Guided Optimal Transport for Multi-Reward Alignment

Text-to-image generation models have achieved remarkable progress in preference optimization, yet achieving robust alignment across diverse reward models remains a significant challenge. Existing multi-reward fusion approaches rely on weighted summation, which is costly to tune and insufficient for balancing conflicting objectives. More critically, optimization with reward models is highly susceptible to reward hacking, where reward scores increase while the perceived quality of generated images deteriorates. We demonstrate that optimizing against a unified global target under heterogeneous reward upper bounds can induce reward hacking, a risk further exacerbated by the inherent instability of weak reward models. To mitigate this, we propose a Pareto Frontier-Guided Optimal Transport (PG-OT) framework. Our method constructs a prompt-specific Pareto frontier and maps dominated samples toward it via distribution-aware optimal transport. Furthermore, we develop both online and offline optimization strategies tailored to diverse reward signal characteristics. To provide a more rigorous assessment, we introduce the Joint Domination Rate (JDR) and Joint Collapse Rate (JCR) as principled metrics to quantify multi-reward synergy and reward hacking. Experimental results show that our approach outperforms strong baselines with an 11% gain in JDR and achieves a near 80% win rate in human evaluations.

Modern text-to-image systems can generate highly realistic images from written descriptions, but improving these systems is difficult because they must satisfy many different goals at once, such as image quality, human preference, and text alignment. Existing approaches usually combine these goals using manually chosen weights, which is hard to tune and can often lead to unstable behavior. In some cases, models learn to artificially increase reward scores while actually producing worse images, a problem known as “reward hacking.” In this work, we show that reward hacking becomes especially severe when different rewards have inconsistent limits or when some reward models are unreliable. To address this issue, we propose a new training framework inspired by the concept of Pareto optimality, which seeks balanced improvements across multiple objectives instead of over-optimizing a single one. Our method guides generated images toward better trade-offs among rewards using an optimal transport strategy that adapts to each prompt individually. We also introduce new evaluation metrics to measure whether different rewards truly improve together and to detect hidden failures during optimization. Experiments demonstrate that our method produces more reliable and higher-quality image generation results than existing approaches, achieving significantly better multi-objective performance and strong human preference ratings.