Who can we trust? LLM-as-a-jury for Comparative Assessment

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the \emph{LLM-as-a-jury} setting and propose BT-$\sigma$, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that \textit{BT-$\sigma$} consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that \textit{BT-$\sigma$} can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

Large language models (LLMs) are increasingly used to judge the quality of text produced by other AI systems, such as summaries or chatbot responses. However, different LLM judges often disagree, exhibit biases, or make inconsistent decisions. Existing approaches typically combine multiple judges by treating them as equally reliable, which can reduce the quality of the final evaluation. In this work, we study a multi-judge ("LLM-as-a-jury") setting and introduce \emph{BT-$\sigma$}, a method that automatically estimates how reliable each LLM judge is without requiring human labels. We first show that LLM judges frequently make inconsistent comparisons, which limits existing aggregation methods. \emph{BT-$\sigma$} learns which judges tend to make more reliable decisions and gives them greater influence when combining opinions. We evaluate our method on several language-generation benchmarks and find that it consistently improves agreement with human judgments compared with existing approaches. These results suggest that accounting for differences in judge reliability can produce more accurate and reliable AI evaluation systems.