Quantifying Biases in LLM-as-a-Judge Evaluations

The evaluation of large language models (LLMs) is increasingly performed by other LLMs, a setup commonly known as "LLM-as-a-judge", or autograders. While autograders offer a scalable alternative to human evaluation, they are not free from biases (e.g., favouring longer outputs or generations from their own model family). Here we propose a statistical framework based on Bayesian generalised linear models (GLMs) that enables researchers to address their primary research questions (e.g., LLM capability or risk assessment), while simultaneously identifying, quantifying and mitigating various biases in their autograders. Our approach can be applied to various evaluation formats (e.g., absolute scores or pairwise preferences) and augments traditional metrics (e.g., inter-rater agreement) by providing precise uncertainty estimates and clarifying sources of disagreement between graders. This framework also enables efficient counterfactual simulations without costly re-evaluation (e.g., assessing agreement after removing systematic biases). We demonstrate these capabilities through simulated examples, with all methods available in an open-source software package. Overall, we introduce a novel framework for autograder evaluation which allows researchers to detect, quantify and correct for various biases in a systematic way.

Automated AI evaluation often relies on AI models to grade other AI models. To trust these “AI judges,” researchers must also evaluate the judges themselves, which typically requires separate experiments. Here, we propose a unified statistical framework for evaluating AI models and their judges simultaneously. Alongside model evaluation, our method quantifies judge biases, identifies sources of disagreement between AI and human judges, and assesses judgment reliability. This reduces the risk of drawing misleading conclusions from biased or unreliable AI judges and supports more transparent automated evaluation.