Conditional Coverage Diagnostics for Conformal Prediction

Evaluating conditional coverage remains one of the most persistent challenges in assessing the reliability of predictive systems. Although conformal methods can give guarantees on marginal coverage, no method can guarantee to produce sets with correct conditional coverage, leaving practitioners without a clear way to interpret local deviations. To overcome sample-inefficiency and overfitting issues of existing metrics, we cast conditional coverage estimation as a classification problem. Conditional coverage is violated if and only if some classifier can achieve lower risk than the target coverage. Through the choice of a (proper) loss function, the resulting risk difference gives a conservative estimate of natural miscoverage measures such as L1 and L2 distance, and can even separate the effects of over- and under-coverage, as well as handle non-constant target coverages. We call the resulting family of metrics excess risk of the target coverage (ERT). We show experimentally that the use of modern classifiers provides much higher statistical power than simple classifiers underlying established metrics like CovGap. Additionally, we use our metric to benchmark different conformal prediction methods. Finally, we release an open-source package for ERT as well as previous conditional coverage metrics. Together, these contributions provide a new lens for understanding, diagnosing, and improving the conditional reliability of predictive systems.

When artificial intelligence systems make predictions, it is crucial they also communicate their uncertainty by providing a range of plausible outcomes. Current techniques can guarantee these ranges are reliable on average across a whole population. However, being right on average does not mean the system is equally reliable for everyone; it might be highly accurate for one group but fail frequently for another. Measuring this localized reliability, known as "conditional coverage," has historically been very difficult. In our paper, we introduce a new way to test whether a model is equally reliable across all specific situations. We turn the evaluation into a simple prediction game: we train a secondary model to guess when the original model's confidence ranges will fail. If this secondary model can predict failures better than a baseline guess, it exposes hidden reliability gaps in the original model. Based on this insight, we created a new family of metrics called ERT to accurately measure these discrepancies, and we provide open-source software to help developers diagnose and improve the trustworthiness of their AI systems.