Robust Bayes-Assisted Conformal Prediction

Bayes--assisted conformal prediction combines the strengths of Bayesian modelling with exact, distribution--free frequentist coverage guarantees. Although conformal validity is preserved even when the Bayesian working model (BWM) is misspecified, the size of the resulting prediction sets can degrade substantially when the prior is poorly aligned with the observed data. We address this limitation by introducing \textbf{RoBAS} (\textbf{Ro}bust \textbf{B}ayes-\textbf{A}ssisted \textbf{S}hrinkage): a Bayes--assisted framework for constructing robust nonconformity scores, with two instantiations: one induced by a heavy--tailed BWM, and a closed--form empirical Bayes shrinkage score. The resulting scores adapt to the quality of the working information encoded in the prior: when this information is reliable, they exploit it to produce efficient prediction sets; when it is weak or inaccurate, they revert to the Distance--To--Average (DTA) score, a robust non--informative baseline. We evaluate the proposed scores on tabular and image regression tasks where the training distribution may differ from the calibration and test distributions, while the calibration and test data themselves remain exchangeable. We find that they are competitive with widely used scores in the absence of such shift, while substantially reducing interval widths in shifted settings.

When AI systems make predictions, it matters not just what they predict, but how confident to be. Conformal prediction is a technique that wraps any AI model's output in a guaranteed prediction interval - a range that is mathematically assured to contain the true answer a specified fraction of the time, no matter how the underlying model was built. One way to make these intervals tighter and more useful is to incorporate background knowledge via Bayesian statistics. The catch: if that background knowledge is wrong - for instance, when a model is deployed on data that looks different from what it was trained on - the intervals can become uselessly wide, defeating the purpose. We introduce RoBAS, a method that automatically adapts to the reliability of the underlying model. When the model performs well, RoBAS exploits the background knowledge to produce tight, informative intervals. When the model struggles - such as under distribution shift between training and deployment - RoBAS detects this from the data and falls back to a robust, data-driven approach, preventing unnecessary inflation of interval widths.