Learning from Comparison: Constrained Projection Policy Optimization for Pareto-Front Improvement

Constrained multi-objective reinforcement learning aims to discover a diverse set of feasible trade-offs, yet scalarization and signed, normalized group-relative advantages can be brittle under objective-scale drift, near-ties, and feasibility scarcity. We propose constrained projection policy optimization (CoPro), which alternates between an E-step moment projection and an M-step policy projection. In the E-step, we solve a Kullback-Leibler (KL)-regularized, moment-constrained projection over each sampled group to compute a nonnegative reweighting distribution ($ q^* $) that promotes feasible Pareto-front (PF) progress, preserves feasibility anchors, and suppresses ambiguous near-ties. This E-step admits a closed-form exponential-family solution and guarantees strictly positive probability mass on feasible anchors whenever feasible candidates appear in the group. In the M-step, we project the policy toward $ q^* $ via weighted maximum likelihood with a trust-region regularizer, yielding a PF-aligned update direction from comparisons without hand-crafted reward shaping. Empirically, CoPro improves feasible PF quality and robustness on constrained multi-objective benchmarks for large language model tool use and analog circuit design tasks; code is available at \href{https://anonymous.4open.science/r/CoPro-8A95/README.md}{\texttt{CoPro}}.

Many real-world AI systems must optimize several goals at the same time while also satisfying hard requirements. For example, an analog circuit design should be fast, power-efficient, and compact, but it must also remain stable; a language model tool-use response should be correct, well formatted, and not overly long. Existing reinforcement learning methods often combine these goals into one hand-designed reward, but this can be fragile when objectives have different scales or when feasible solutions are rare. This paper introduces CoPro, a method that learns from comparisons among candidates in the same group and gives more weight to candidates that improve the feasible Pareto front. Experiments on tool-calling tasks and analog circuit design benchmarks show that CoPro improves feasible trade-offs, reduces constraint violations, and produces more stable optimization behavior. This makes it useful for optimization problems where success requires both high performance and reliable constraint satisfaction.