Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better

Conformal prediction (CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick (PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction

Machine learning systems are increasingly used to support decisions in areas such as healthcare and finance, where it is important to know not only what a model predicts, but also how uncertain that prediction is. Conformal prediction is a popular way to provide such uncertainty estimates, often in the form of a prediction interval or set that is expected to contain the correct answer with a specified probability. These methods are commonly evaluated by two numbers: how often the answer is covered, and how short the prediction interval is. Our work shows that these two numbers alone can be misleading. We introduce a simple randomized construction, called the Prejudicial Trick (PT), that can make prediction intervals look shorter on average while still maintaining the required coverage. However, this improvement is not meaningful: the same input may receive very different prediction intervals across repeated runs, and some users may receive uninformative predictions purely due to randomness. To expose this failure mode, we propose interval stability, a diagnostic that measures how much prediction sets vary across repeated runs. Our findings suggest that uncertainty estimation methods should be evaluated not only by coverage and size, but also by whether their predictions are stable and reliable for individual users.