The Sign Estimator: Preference Modeling for LLM Alignment under Heterogeneity

Traditional large language model (LLM) alignment methods are based on Reinforcement Learning From Human Feedback (RLHF), which learns a single reward model (implicitly or explicitly) from pairwise comparison data. This approach implicitly assumes homogeneous preferences across human labelers---an assumption that is violated in practice. As a result, the learned reward model in RLHF is generally misspecified: Prior work shows that it is inconsistent with the population-average utility, incurring large distortion, and that recovering this utilitarian objective is provably impossible in the worst case. In this work, we show that the average utility is recoverable under a mild assumption. Our method, the Sign Estimator, simply replaces the standard cross-entropy loss function with a notion of binary classification loss and yields a reward model that is ordinally consistent with the population-average utility. We further establish a fast finite-sample convergence rate of $O({n^{-{1}/{3}}})$ , which provides, to our knowledge, the first consistent estimator for heterogeneous preferences that does not suffer from the curse of dimensionality.

AI assistants are often trained using human feedback: people compare two possible answers, and the system learns which kinds of answers people prefer. A common way to combine this feedback effectively treats all people as if they shared one single preference. In reality, people disagree, sometimes strongly. When these disagreements are ignored, the learned system can end up reflecting a distorted version of what the overall population prefers. This paper studies how to better combine feedback from people with different preferences. We show that the standard approach can give too much influence to uncertain or weak preferences and too little influence to strong ones. We propose a simple alternative, called the Sign Estimator, that focuses on which answer is preferred rather than trying to fit the exact strength of each preference. This small change can be used in existing feedback-training pipelines without changing the model architecture. We prove that, under natural conditions, this method recovers the direction of the population’s average preference. We also test it on real human-feedback datasets and simulated diverse user populations, where it predicts preferences more accurately than the standard method and several approaches that explicitly model different user groups.