Why Hierarchical Structure Matters for Uncertainty Quantification in Agentic AI Evaluation

Uncertainty quantification is critical for trustworthy AI evaluations, yet the statistical methods commonly used were designed for data-rich settings and largely ignore the hierarchical structure of evaluation data. Here, we compare four uncertainty quantification methods that differ in how much they account for evaluation data structure ("naive" confidence intervals, item-level bootstrap, cluster bootstrap, and hierarchical Bayesian GLMs), across QA benchmarks (use cases 1 and 2) and an agentic benchmark (use case 3), where we additionally assess their reliability in low-data regimes. Across both setups, methods that do not account for hierarchy produce biased performance estimates that may lead to spurious inferences of significant differences between models. The cluster bootstrap accounts for hierarchy but its point estimates shift substantially across data regimes, limiting its reliability. Hierarchical Bayesian GLMs produce less biased estimates in QA benchmarks and the most stable estimates in the agentic benchmark, with reasoning effect estimates 4x less sensitive to data regime than the cluster bootstrap. These findings demonstrate that accounting for hierarchical data structure is essential for reliable uncertainty quantification in AI evaluation. Despite being well-established in the natural and social sciences, hierarchical GLMs remain largely absent from AI evaluation practice; to encourage their adoption, we implement all models in the open-source HiBayES package.