Context-Aware Conformal Prediction for VLM-Based Driving Scene QA

Vision-Language Models (VLMs) deployed in autonomous driving must provide reliable uncertainty estimates for safety-critical scene understanding. When a model is unsure, rather than returning a single answer that may be wrong, Conformal Prediction (CP) returns a small set of candidate answers that is guaranteed to contain the correct one with high probability, without distributional assumptions. However, standard CP only guarantees this coverage on average across all inputs (marginal coverage), and can severely under-cover safety-critical subgroups such as rare perception questions or high-risk driving scenes. We propose Context-Aware Conformal Prediction (CA-CP), which integrates autonomous driving planner features, specifically planner uncertainty and temporal stability in planned trajectories, as external calibration signals for VLM-based driving scene Question Answering (QA). Evaluated on 16,655 WaymoQA pairs across six VLMs, CA-CP improves worst-slab conditional coverage along the VLM uncertainty axis on all 6 VLMs and increases average worst-group coverage. Among the delivery methods evaluated, KNN kernel-weighted quantiles achieve the best conditional coverage with only a modest increase in prediction set size. These findings demonstrate that planner signals provide complementary difficulty information that enhances conditional coverage beyond VLM-internal features alone.