Margin-Adaptive Confidence Ranking for Reliable LLM Judgement

Jung et al. (2025) introduce a hypothesis testing framework for guaranteeing agreement between large language models (LLMs) and human judgments, relying on the assumption that the model’s estimated confidence is monotonic with respect to human-disagreement risk. In practice, however, this assumption may be violated, and the generalization behavior of the confidence estimator is not explicitly analyzed. We mitigate these issues by learning a dedicated confidence estimator instead of relying on heuristic confidence signals. Our approach leverages simulated annotator diversity and a margin-based ranking formulation to explicitly model how confidently an LLM distinguishes between human-agreement and human-disagreement cases. We further derive generalization guarantees for this estimator, revealing a margin-dependent trade-off that informs the design of an adaptive estimator training procedure. When integrated into fixed-sequence testing, the learned confidence estimator yields improved ranking accuracy and empirically strengthens the monotonic relationship between confidence and disagreement risk, leading to higher success rates in satisfying target agreement levels across multiple datasets and judge models.

Large language models are increasingly used to judge whether generated answers agree with human preferences, but their confidence scores are often unreliable. This makes it difficult to know when an LLM-based judgment should be trusted. We propose a learned confidence estimator that is trained to distinguish cases where an LLM is likely to agree with humans from cases where disagreement is more likely. Our method accounts for diversity in human judgments and learns to rank more reliable decisions above less reliable ones. We also provide analysis showing when this estimator can generalize to new examples.