A Close Look at Negative Label Guided Out-of-distribution Detection in Pre-trained Vision-Language Models

Advances in pre-trained vision-language models have enabled zero-shot out-of-distribution (OOD) detection using only in-distribution (ID) labels. Recent methods in this direction expand the label space with negative labels to enhance the discrimination between ID and OOD inputs. Despite their promising progress, there remains a limited understanding of their empirical effectiveness in open-world scenarios, where negative labels can arbitrarily diverge from real OOD ones. This paper bridges this research gap with the helm of a novel energy-based framework, where the energy function is built upon the margin between the similarity of an input to ID labels and that to negative labels. Guided by this framework, we prove that the inherent tolerance of such methods to the sampling bias essentially stems from estimating the worst-case energy function over a KL-constrained set of potential distributions centered on the negative label distribution. Furthermore, our theoretical analysis reveals that existing methods suffer from over-pessimism and consequently high sensitivity to outliers. Provably, we can alleviate these problems by leveraging Rényi divergence to refine potential distributions. Extensive experiments empirically manifest that our method establishes a new state-of-the-art across a variety of OOD detection settings.

Open-world vision systems must reliably detect inputs from unseen classes, but current CLIP-based OOD detectors using negative labels lack a clear theoretical explanation and can be brittle when sampled negative labels diverge from real OOD data. We study negative-label-guided OOD detection through a distributionally augmented energy-based framework, showing that existing methods implicitly estimate a worst-case energy function over a KL-constrained set of possible negative-label distributions. This explains their robustness to sampling bias, but also reveals an over-pessimistic weighting mechanism that makes them sensitive to outlier or false negative labels. To address this, we replace the KL-based uncertainty set with a Rényi-divergence formulation, producing a milder, polynomial-bounded worst-case distribution with tunable conservatism.