Efficient Inference for Noisy LLM-as-a-Judge Evaluation

Large language models (LLMs) are increasingly used as automatic evaluators of generative AI outputs, a paradigm often referred to as "LLM-as-a-judge." In practice, LLM judges are imperfect predictions for the underlying truth and can exhibit systematic, non-random errors. Two main approaches have recently been proposed to address this issue: (i) direct measurement-error correction based on misclassification models such as Rogan--Gladen-style estimators, and (ii) surrogate-outcome approaches such as prediction-powered inference (PPI), which correct bias by calibrating prediction residuals on a small set of gold-standard human labels. In this paper, we systematically study the performance of these two approaches for estimating mean parameters (e.g., average benchmark scores or pairwise win rates). Leveraging tools from semiparametric efficiency theory, we unify the two classes of estimators by deriving explicit forms of efficient influence function-based efficient estimators and characterize conditions under which PPI-style estimators attain strictly smaller asymptotic variance than measurement-error corrections. We verify our theoretical results through simulations and demonstrate the methods on a real-data example using our open-source software package for performing the calibration.

Evaluating modern AI systems is expensive: it usually requires humans to grade thousands of model outputs. To save effort, researchers increasingly use large language models as automatic graders — a practice called "LLM-as-a-judge." But these AI graders aren't perfectly accurate, and their mistakes tend to be biased rather than random, so naively averaging their scores gives a misleading picture of how good an AI system really is. Two statistical fixes have been proposed: one directly models the grader's error rates and adjusts the scores; the other uses a small batch of trusted human labels to calibrate the AI grader's outputs. We build a unified framework that puts both on the same footing, and pinpoint when each works best — showing that the calibration-based approach is often, though not always, more reliable for the same human labeling budget. We confirm this in simulations and on a real benchmark, and release an open-source package so others can apply the methods to their own evaluations.