Human-AI Collaborative Uncertainty Quantification

AI predictive systems increasingly support high-stakes decision making, yet robust decisions under uncertainty often rely on human capabilities beyond AI alone. This motivates collaborative approaches that combine human judgment with AI predictions. We study this problem through the lens of uncertainty quantification and introduce Human-AI Collaborative Uncertainty Quantification, a framework in which an AI system refines a human expert’s proposed prediction set subject to two principles: counterfactual harm, requiring that the AI not degrade correct human judgments, and complementarity, requiring recovery of correct outcomes the human missed. At the population level, we show that the optimal collaborative prediction set has a simple two-threshold structure over a single score function, governing pruning and augmentation relative to the human proposal. Building on this characterization, we develop offline and online calibration algorithms with distribution-free finite-sample guarantees. The online algorithm adapts to arbitrary distribution shifts, including settings where human behavior evolves through interaction with the AI. Empirically, we show that collaborative prediction sets outperform human-only and AI-only baselines, achieving improved coverage--efficiency tradeoffs across image classification, regression, and text-based medical decision making.

AI predictive systems increasingly support high-stakes decision making, yet robust decisions under uncertainty often rely on human capabilities beyond AI alone. This motivates collaborative approaches that combine human judgment with AI predictions. A doctor brings experience and context the AI lacks, while the AI can spot patterns across enormous amounts of data. But how can we combine them so the team is more trustworthy than either one alone? We study a setting where, instead of committing to a single answer, each side offers a shortlist of plausible answers. The human proposes their shortlist first, and our method lets the AI refine it: trimming options that are likely wrong and adding correct ones the human overlooked. We hold the AI to two rules: never make the human's shortlist worse (counterfactual harm guarantee), and add value by catching what the human missed, and we prove these guarantees hold mathematically, even as the situation shifts, such as when the human gradually changes its interactive behavior and learns to rely on the AI. Across a variety of tasks including medical diagnosis and image recognition, these collaborative shortlists were both more reliable and more concise than the human's or the AI's alone, offering a principled recipe for human–AI teamwork where trust matters most.