Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine seven LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliability of LLM-as-a-Judge and identifying potential causes of unreliability.

AI systems are increasingly used to judge the quality of other AI systems' outputs, a setup often called "LLM-as-a-Judge." Judging this way is fast and inexpensive, but it raises a fundamental question: Can we trust the judgments of LLM-as-a-Judge? Existing methods mostly examine whether an AI judge's scores align with human scores. However, agreement in scores does not reveal whether the judge itself is sound, that is, whether it judges in a stable and principled manner rather than producing numbers that coincide with human ratings. Our central idea is to regard an AI judge not as an system that merely outputs scores, but as a measurement instrument, comparable to an exam, whose own reliability must be established before its readings can be trusted. We treat this as fundamentally a measurement problem and address it using methods from the social sciences. We adapt that methodology to AI judges. Specifically, we treat an AI judge as an exam and its judgments as exam results, which allows us to examine two properties separately. First, we assess whether the judge remains consistent when its instructions are slightly reworded, since a sound instrument should not yield different readings merely because the wording has changed. Second, we assess whether it perceives quality in the same way that people do. Testing seven widely used AI models, we found that no model is reliable across all settings, and that the causes of failure differ. Some models are sensitive to the exact wording of their instructions, while others perceive quality on a different scale than humans, or appear to assess something humans do not. Crucially, our framework does not merely detect that a judge is unreliable; it identifies which of these causes is responsible. This distinction matters because different causes call for different remedies. When a judge is unstable under rewording, we find that giving it more detailed evaluation instructions improves its consistency. When a judge's perception of quality diverges from that of humans, adjusting the rating scale it uses can bring its measurements closer to human judgment. By pinpointing the cause and pointing toward an appropriate adjustment, our framework offers practitioners a practical means of diagnosing an AI judge, and improving it, before relying on its judgments.