Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the *distortion* — defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility — can scale exponentially with the Bradley-Terry temperature parameter $\beta$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($\mu$) and the KL reference policy ($\pi_{\mathrm{ref}})$. To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley–Terry model, the distortion is $\tilde{\Theta}(\beta B + \beta)$, where $B$ is an upper bound on the log density ratio between $\mu$ and $\pi_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $\mu = \pi_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(\beta)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $\mu$.

Reinforcement Learning from Human Feedback (RLHF) is widely used to align large language models with human preferences and values. However, RLHF is designed as if there were a single representative user. This has led to concerns that, when users have diverse preferences, RLHF may fail to produce models that are useful on average across the whole user population. In this work, we focus on the mismatch between the distribution from which training data are sampled and the distribution of outputs from the model before RLHF. We then theoretically analyze how well RLHF optimizes average usefulness compared with the optimal value, a quantity known as distortion. Our results have two main parts. First, when these two distributions match, RLHF can produce a model that is nearly as useful on average as the best possible model. Second, when there is a mismatch, some loss is unavoidable, but the loss is relatively moderate unless the mismatch is extreme. These findings suggest that RLHF does not usually break down simply because users have diverse preferences, and they provide theoretical support for using RLHF.