Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge

Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate $n$ independent thinking--rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley–Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.

Large language models are increasingly used to judge the quality of other AI systems, for example by deciding which of two answers or translations is better. However, a single judgment from such a model can be noisy, and the problem becomes harder when the correct answer may be that the two options are effectively tied. A common solution is to ask the model several times and take a majority vote, but this can ignore important information: a narrow split among votes is not the same as a strong consensus. This paper studies how to combine repeated model judgments more reliably. We propose a lightweight calibration method that looks at the full pattern of votes, including how often the model chooses each side and how often it declares a tie. Using a small set of labeled examples, the method learns how to interpret the model’s voting behavior and then makes a final decision that better matches the evaluation goal. Across machine translation and reward-model evaluation benchmarks, this approach produces more reliable judgments than standard voting-based methods. The results suggest that automated evaluation can be improved substantially by carefully aggregating repeated model judgments, while still treating human review as important in high-stakes settings.