Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.

As large language models become more widely used, researchers need ways to compare them and decide which ones perform best. There are currently two methods: comparing each model's answers to a known correct answer, or showing two models' answers side-by-side and picking the better one. A big worry is that side-by-side comparisons may be biased — an answer might be preferred not because it's correct, but because it simply looks correct. We tested the side-by-side method on questions where the correct answer is known, and found that both approaches order the models in nearly the same way. This suggests that side-by-side comparisons are more reliable than recent debates have suggested. While this approach could make model evaluation cheaper and more flexible, more work is needed to ensure these systems can't be easily tricked or biased.