SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose SCOPE (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $\alpha$. To supply SCOPE with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while SCOPE consistently satisfies the target risk bound (empirical FDR $\approx 0.097$--$0.099$ at $\alpha=0.10$) and retains substantial coverage. Compared to vanilla baselines, SCOPE accepts up to $2.4\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.

Large language models (LLMs) such as GPT or Llama are increasingly used as automatic judges that pick the better of two answers, replacing expensive human annotators in everything from model evaluation to building training datasets and ranking systems on public leaderboards. But these AI judges are imperfect: they have systematic biases (for example, they tend to favor whichever response is shown first), and they often report high confidence even when they are wrong. Without a way to know which of their decisions to trust, errors silently propagate into the rankings, reward models, and datasets built on top of them. We introduce SCOPE, a method that lets a user pick an acceptable error rate (for example, "I am willing to tolerate at most 10% wrong decisions among those I accept") and then automatically determines which judgments meet that bar and which should be flagged as uncertain and rejected. To make this work, we also propose Bidirectional Preference Entropy (BPE), a simple technique that asks the judge to evaluate the same pair of answers twice in opposite orders and combines the two responses to cancel out the position bias. Across standard benchmarks and a range of popular LLM judges, SCOPE keeps the error rate within the user's chosen budget while accepting substantially more judgments than baseline calibration approaches, making LLM-as-judge a more trustworthy tool for its growing role in AI evaluation and alignment.