CAOS: Conformal Aggregation of One-Shot Predictors

One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.

Some AI systems can adapt to a new task after seeing only one labeled example, but they rarely know how to say, “I’m not sure.” We want these systems to return a short list of plausible answers that contains the correct answer most of the time. This is especially important when labels are scarce, such as when clinicians annotate a few patient images with the landmarks an AI system should find. The same examples are needed for two jobs: helping the model make predictions and checking whether its answer lists can be trusted. Standard approaches often force a choice between these uses. We introduce CAOS, a method that lets every labeled example help with both jobs. CAOS also accounts for the fact that some examples are more relevant to a new case than others, so it gives more influence to examples that better support a candidate answer. The result is a shorter, more useful answer list with a mathematical promise that the correct answer is included most of the time, when future cases resemble the labeled examples. In experiments on facial landmarking and text classification, CAOS kept this reliability promise while producing shorter, more informative answer lists than standard methods.