Optimal Transport for LLM Reward Modeling from Noisy Feedback

Reward models are fundamental to Reinforcement Learning from Human Feedback (RLHF), yet real-world datasets are inevitably corrupted by noisy preference. Conventional training objectives tend to overfit these errors, while existing denoising approaches often rely on homogeneous noise assumptions that fail to capture the complexity of linguistic preferences. To handle these challenges, we propose SelectiveRM, a framework grounded in optimal transport. We first devise a Joint Consistency Discrepancy to align the distribution of model predictions with preference data. Furthermore, to address the limitation of strict mass conservation which compels the model to fit outliers, we incorporate a Mass Relaxation mechanism via partial transport. This enables the autonomous exclusion of samples with noisy preference that contradict semantic consistency. Theoretically, we demonstrate that SelectiveRM optimizes a tighter upper bound on the unobserved clean risk. Extensive experiments validate that our approach significantly outperforms state-of-the-art baselines across diverse benchmarks.

Large language models are often trained to give better answers by learning from human preferences, such as which of two responses people like more. However, these preference labels are not always reliable. Human annotators may disagree, become tired, or make mistakes, and automatic judges can also give incorrect feedback. If a model learns directly from these flawed labels, it may copy the mistakes instead of learning what people truly prefer. This paper introduces SelectiveRM, a method for training reward models more reliably when preference data contains errors. The key idea is to compare the model’s judgments with the overall pattern of the data, while allowing the training process to ignore examples that appear inconsistent or misleading. In this way, the model can focus more on useful feedback and less on noisy feedback. Experiments on several benchmarks show that SelectiveRM produces better reward models than existing methods, making it a promising step toward safer and more reliable language model training.