CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders---such as verbosity, stylistic preferences, or training artifacts---causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%.

Many researchers now use AI models to help evaluate the answers produced by other AI systems. A common approach is to ask several AI “judges” for scores and then average their opinions, much like taking a vote. This can save time and money compared with human evaluation, but it has an important weakness: the judges may make the same kinds of mistakes. For example, several judges might give higher scores to longer answers, more formal writing, or answers that look technical, even when those features do not actually make the answer better. This paper introduces CARE, a method for combining the scores from multiple AI judges more reliably. Instead of assuming that each judge makes independent errors, CARE looks for shared patterns that may be influencing several judges at once. It then tries to separate the actual quality of an answer from these misleading signals. Across a range of evaluation tasks, CARE produces more accurate combined scores than standard approaches such as voting or averaging. The method can also help diagnose what kinds of superficial features may be affecting AI judges, making automated evaluation more trustworthy and easier to inspect.