Calibrated Preference Learning: The Case of Label Ranking

Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.

Probabilities play an important role in modern AI systems because they express a model's confidence in its predictions, and these confidence estimates should reflect reality for AI systems to be trustworthy. This problem is studied in the field of calibration. We investigate calibration for preference learning, where models learn to rank items, such as movies, products, or AI-generated responses, based on human preferences. Our paper introduces several ways to define calibration for ranking models and shows theoretically that these notions capture different aspects of reliability. We also find that many widely used ranking and reward models are poorly calibrated, even when they achieve high accuracy. Our findings lay the theoretical groundwork for developing more trustworthy preference-learning systems in the future.