Distribution-Free Uncertainty for Continuous Agent Evaluation

We adapt split conformal prediction and adaptive conformal inference (ACI) to continuous AI agent evaluation, providing distribution-free coverage guarantees for forecasted quality scores. Conformal intervals achieve calibration error below 0.02 across all nominal levels at the 24h horizon, while ACI correctly widens intervals by 35\% following agent releases then reconverges. We further develop compositional uncertainty bounds for multi-agent pipelines (validated via simulation across inter-stage correlations $\rho \in [-0.5, 0.9]$), a conformal abstention rule for pairwise rankings with controlled false-ranking rate, and FDR-corrected abstention for leaderboard-scale multiple testing. Evaluating 50 agents via 18 real-time signals collected hourly, we show that per-agent conditional coverage is well-concentrated around the nominal level (mean 80.4\%, 90\% of agents within $[72\%, 90\%]$), and that cross-source sentiment divergence predicts ranking instability ($r{=}0.64$, $p{<}0.01$). A circularity-controlled validation confirms the framework captures signal beyond benchmarks ($\rho_s{=}0.52$, $p{<}0.01$, $n{=}35$). Code and data are released under CC BY 4.0.