Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching

Vision-Language Models (VLMs) are typically deterministic in nature and lack intrinsic mechanisms to quantify epistemic uncertainty, which reflects the model’s lack of knowledge or ignorance of its own representations. We theoretically motivate negative log-density of an embedding as a proxy for the epistemic uncertainty, where low-density regions signify model ignorance. The proposed method REPVLM computes the probability density on the hyperspherical manifold of the VLM embeddings using Riemannian Flow Matching. We empirically demonstrate that REPVLM achieves near-perfect correlation between uncertainty and prediction error, significantly outperforming existing baselines. Beyond classification, we also demonstrate that the model also provides a scalable metric for out-of-distribution detection and automated data curation.

Modern AI systems like CLIP can match images with text. But these systems have a blind spot: they answer with the same apparent confidence whether the input is clear and familiar or blurry, nonsensical, and unlike anything they've seen before. They cannot say "I don't really know." We developed REPVLM, a method that gives these AI systems a sense of self-doubt. Our key insight is that when the AI processes a familiar input, it places it in a "crowded neighborhood" of its internal map, surrounded by similar things it has seen during training. Unfamiliar inputs land in sparse, empty regions. By measuring how crowded the neighborhood is around each input, we get a reliable score for how much the AI actually knows what it's looking at. This matters because AI is increasingly used in settings where overconfident wrong answers are costly. REPVLM can flag unreliable predictions, detect unfamiliar inputs, and automatically clean noisy data, making AI systems safer and more trustworthy.