A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates—more data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.

Large language models are often evaluated by asking other LLMs to judge their answers, but judge models differ in how reliably they distinguish between answers. Treating all judges equally can distort rankings and make weak evidence look overly certain. We study how to rank LLMs from pairwise comparisons when judges have different discrimination abilities. Our method jointly estimates each candidate model’s quality and each judge’s reliability, so informative judges receive more weight and noisy judges receive less, without requiring human ground-truth labels. It also provides uncertainty estimates for ranking differences. This makes LLM-as-a-judge evaluation more reliable and useful for realistic model comparison.