Hybrid Adaptive Prediction Sets for Conformal Prediction

Probability-based Conformal Prediction (CP) often fails under distribution shift due to deep neural network overconfidence, producing unreliable prediction sets. To address this, we introduce Hybrid Adaptive Prediction Sets (HAPS), a non-conformity framework combining the discriminative precision of supervised classifiers with the zero-shot geometric robustness of frozen Foundation Models (FMs). By multiplicatively fusing parameterized probabilities and semantic latent similarities, HAPS enforces a ``geometric veto'' to systematically suppress overconfidence. We propose three scalable variants ($k$NN-HAPS, Fix-HAPS, and Var-HAPS) to balance topological precision and computational cost. Evaluations on ImageNet-1K show HAPS drastically reduces average prediction set sizes while maintaining exact marginal coverage. Furthermore, HAPS significantly outperforms standard baselines under severe shifts (ImageNet-A/R), producing better coverage and efficiency performances.