Answer, Clarify, or Abstain: Fine-Grained Selective Prediction for Medical VLMs

Medical vision-language models (VLMs) deployed for clinical QA face a fundamental efficiency problem: always-answering regardless of input quality wastes clinician attention on unreliable outputs. Under realistic deployment perturbations - image corruption, noisy text, or image-text mismatch - models produce confident but incorrect answers that are costly to review and dangerous to act on. We address this with three contributions: a robustness benchmark, a fine-grained selective prediction framework, and a lightweight condition-aware reliability layer. We stress-test four medical VLMs across chest X-ray, CT, and fundus imaging datasets under image corruption, text corruption, and cross-modal mismatch. Performance degrades sharply, and entropy-based uncertainty signals remain weakly correlated with correctness across all conditions. We formalize fine-grained selective prediction: models may answer, request clarification for corrupted inputs, or abstain for genuine uncertainty - routing each case to its most efficient resolution rather than collapsing both failure modes into a single reject option. A condition-aware reliability layer - a small MLP on frozen hidden states - estimates reliability in a single forward pass without backbone modification. It improves ROC-AUC by up to 6-8 points over entropy baselines (including semantic entropy, which requires multiple forward passes), adapting with fewer than 500 labeled samples.