DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC), a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a KL-robust (entropic) satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.

Large language models are often aligned using human preference feedback, but people do not always agree on what makes an answer good. Existing alignment methods usually average these preferences into a single score, which can cause models to choose responses that look good on average but are risky, polarizing, or unsatisfactory for some users. This paper proposes DARC, an inference-time method that selects model responses while explicitly accounting for disagreement and risk. Instead of retraining the model, DARC reranks a set of candidate responses using a robust objective that favors answers with high expected satisfaction and lower uncertainty across users or evaluators. The method also provides practical controls for limiting risk during deployment. Experiments show that DARC reduces disagreement and improves worst-case outcomes while maintaining competitive average response quality, especially on prompts where user preferences are highly divided.