Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

Aligning large language models to human preferences is inherently multidimensional, yet most pipelines collapse heterogeneous signals into a single objective. We seek to answer what it would take to simultaneously align a model across various domains spanning those with: verifiable rewards, non-verifiable subjective preferences, and complex interactive scenarios. Such multi-objective alignment setups are often plagued by individual objectives being at odds with each other, resulting in inefficient training and limited user control during inference. To address these issues, we propose Multi-Action-Head ALignment with PRM-guided DecOding (MAHALO), a unified framework that standardizes PRM training across verifiable and non-verifiable settings for step-level supervision, performs vectorized multi-objective alignment with Multi-Action-Head DPO, and enables controllable inference through objective-specific weighting and PRM-guided decoding. Experiments across math reasoning, human values alignment, and multi-turn tutoring show that MAHALO jointly improves multiple objectives simultaneously with limited interference, while remaining generalizable and adaptable across domains and offering flexible user control at inference time. Our code is available at: https://github.com/pearls-lab/multiobj-align.

AI assistants often need to satisfy several goals at once. An AI tutoring assistant, for example, should give correct answers and keep students engaged. Current training methods often squeeze these goals into one score, hiding trade-offs and making the assistant hard to adjust. This paper introduces a way to train and steer one language model across several goals without training a separate model for each goal. It adds small goal-specific decision layers on top of a shared model, so one layer can focus on accuracy while another focuses on engagement. It also trains “step-by-step judges” that assess parts of an answer, not just the final result, and uses those judgments to guide the model as it writes. In tests on math, human preference data, and tutoring conversations, the approach improved several goals at the same time and let users change the balance among them as the model generates an answer. Checkable goals such as math accuracy benefited especially from search during generation, while subjective goals benefited from multi-goal training. This could help build AI assistants that are easier to control, useful across many domains, and better matched to different people’s needs.