What Does Preference Learning Recover from Pairwise Comparison Data?

Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets $(x, y^+, y^-)$, where response $y^+$ is preferred over response $y^-$ for context $x$. The Bradley--Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency---namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.

Many modern AI systems are trained using comparisons: for a given prompt, a human or another system chooses which of two responses is better. This kind of feedback is easier to collect than asking for precise numerical scores, so it has become a central tool for training language models. A common method assumes that every response has a hidden quality score, and that preferences can be predicted by comparing these scores. But real comparison data may not actually follow this simple scoring model. In this paper, we start from the data itself and ask what preference information is truly contained in a distribution of comparisons. We define this information formally, identify when it can be captured by the standard scoring model, and explain what the model learns when that assumption fails. We also show that learning speed depends on two key properties of the data: how clear the preferences are, and how well the comparisons connect different responses. These results help practitioners understand when standard preference-learning methods are reliable, when they may be misleading, and how to collect comparison data that is easier to learn from.