STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. Majority vote discards annotator reliability and item-level ambiguity, often yielding unstable comparisons across annotator subsets. We introduce STABLEVAL, a disagreement-aware evaluation framework that models latent item correctness and annotator-specific confusion patterns to produce posterior expected item credit and calibrated agent-level scores. Unlike label-denoising approaches such as Dawid--Skene, STABLEVAL is explicitly designed for stable and uncertainty-aware system evaluation rather than hard label recovery. We formalize ranking stability as a first-class evaluation objective and analyze how aggregation methods preserve or distort underlying annotator behavior. Across controlled synthetic experiments and multiple real-world human-annotated benchmarks, majority vote exhibits increasing score error and ranking instability under annotator heterogeneity and adversarial noise, while STABLEVAL yields more stable and statistically grounded system rankings. These results demonstrate that modeling disagreement is essential for robust and reproducible AI evaluation.

How do we evaluate AI systems and decide which one wins? Today, researchers usually ask multiple people to rate AI outputs and pick the most popular answer through majority vote. But people often disagree, and majority vote ignores that. The result is fragile rankings that can shift when you change who's judging. We built STABLEVAL, a framework that treats disagreement as useful information rather than discarding it. Our method learns each judge's tendencies and distinguishes genuinely ambiguous items from those where judges are simply making mistakes. Rather than collapsing every rating into a single label, it tracks our confidence in each judgment and accounts for each judge's tendencies when computing the final scores. Across tests covering chatbot preferences, abuse detection, summarization, and medical evidence review, STABLEVAL produced rankings that stayed consistent even when we swapped out which judges were used, while the majority vote often flipped under the same conditions. As AI systems are increasingly chosen based on human ratings in high-stakes settings like safety and medicine, this makes evaluation more trustworthy.