Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

Out-of-distribution (OOD) detection has emerged as a popular technique to enhance the reliability of machine learning models by identifying unexpected inputs from unknown classes. Recent progress in pre-trained vision–language models (VLMs) has enabled zero-shot OOD detection without access to in-distribution (ID) training data; in this setting, existing methods commonly treat text embeddings of class names as class prototypes. In this paper, we challenge this widely adopted “text-as-prototype” paradigm by theoretically showing that off-the-shelf textual prototypes are generally misaligned with the optimal visual prototypes, yielding an intrinsic \textit{modality gap} that cannot be eliminated by prompt engineering alone. To mitigate this gap under the post-hoc constraint, this paper presents an online pseudo-supervised framework that directly learns class prototypes in the visual feature space using unlabeled test-time data streams and soft predictions from the pre-trained VLMs. We provide theoretical guarantees for the convergence of the online optimization procedure. Extensive experiments empirically manifest that our method achieves a new state of the art across a variety of OOD detection setups.

Modern AI systems are often tested in the messy real world, where they may encounter images from categories they were never trained to recognize. Existing vision-language methods such as CLIP try to detect these unfamiliar inputs by comparing images with text descriptions of known classes, but the paper shows that these text-based “prototypes” do not perfectly match the best visual representations of those classes. The research introduces an online method that learns better class prototypes directly in the image-feature space, using unlabeled test-time data and the model’s own soft predictions as guidance. This lets the system adapt after deployment without retraining the original model or needing labeled examples. By respecting the gap between language and vision representations, the method improves the reliability of out-of-distribution detection across major benchmarks. This matters for safer AI deployment in settings where recognizing “unknown” inputs is as important as classifying familiar ones.