RSA-CP: Efficient Conformal Prediction in Small-Sample Regimes via Random Score Alignment

Conformal Prediction (CP) provides rigorous finite-sample coverage guarantees, yet its statistical efficiency hinges critically on the size of the calibration set. In data-scarce regimes, CP often suffers from volatile quantile estimation, leading to overly conservative and wide prediction intervals. To address this, we propose Random Score Alignment-Conformal Prediction (RSA-CP), a simple framework designed to improve sample efficiency in small-sample CP. Instead of requiring the computationally intensive generation of full synthetic datasets, RSA-CP enhances calibration by directly aligning real scores with a high-resolution reference score distribution. By employing an optimal transport mapping, our framework refines "step-like" quantile increments through a globally optimal use of reference information. We provide theoretical guarantees establishing that RSA-CP maintains robust coverage without any distributional assumptions on the reference scores. Empirical evaluations demonstrate that RSA-CP consistently produces shorter and more precise prediction intervals while maintaining finite-sample coverage guarantees. Overall, RSA-CP offers a computationally efficient and theoretically grounded solution for robust uncertainty quantification under limited data.

Many high-stakes predictive systems need to do more than give a single answer. They also need to express uncertainty: how much should we trust this prediction? Conformal prediction (CP) is a statistical framework designed for this purpose. It produces prediction sets that contain the true answer with high probability, without requiring strong assumptions about the prediction model. CP does this by using calibration data and nonconformity scores, which measure how unusual a new prediction looks compared with past examples. A key difficulty arises when the calibration dataset is small. In this setting, standard conformal prediction has limited resolution and may become overly conservative. Its prediction sets can become much larger than necessary, and in extreme cases may be too broad to be useful. This small-sample limitation is the main problem that motivated our research. We address this problem by introducing Reference-Score Augmented Conformal Prediction (RSA-CP). The main idea is to supplement the limited real calibration scores with additional reference scores. These reference scores may come from a random generator, a fitted model, previous data, or another relevant source. RSA-CP uses this extra information to refine the conformal threshold, but it does so carefully through rank-based correction rules. These rules preserve finite-sample coverage bounds that are easy to compute and remain valid regardless of how the reference scores are chosen. When the reference scores are informative, meaning that they are distributionally close to the real calibration scores, RSA-CP can produce smaller and more useful prediction sets than standard split conformal prediction. This research matters because reliable uncertainty quantification is especially important when data are limited but decisions still need to be made carefully. Such situations arise in scientific studies and medical applications. RSA-CP shows that external reference information can make conformal prediction more informative without giving up its core finite-sample distribution-free guarantees. In this way, the method helps make uncertainty quantification more useful in small-sample prediction problems.