Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness

Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the "Hubris of Semantics", where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this Open-World Trustworthiness Paradox, we propose Immuno-VLM, a bio-inspired framework that adapts the biological principle of Immunological Negative Selection to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate "Semantic Antibodies", textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes. Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.

Modern vision-language AI systems can recognize images using natural language labels, and they often perform well even on images that look different from their training examples. However, these systems can still be overconfident when they see something truly unfamiliar. For example, an object that only resembles a known class, such as a robotic dog or an unusual texture, may still be confidently labeled as a normal dog. This can be risky in real-world applications where the system should know when to say "I am not sure." This paper proposes Immuno-VLM, a method inspired by how the immune system protects the body. Instead of only learning what known objects look like, the method also creates useful examples of "nearby unknowns." A language model is used to imagine objects or situations that are similar to known classes but should not be accepted as them, such as look-alike animals or impossible contexts. These imagined examples help the vision model build safer boundaries around known categories. The method then trains a small additional component that keeps normal images close to their correct categories while pushing them away from these imagined unknowns. At test time, the system compares how much an image resembles known classes versus these warning examples. This allows it to either make a normal prediction or reject the image as unfamiliar. Experiments show that this approach improves the reliability of open-world image recognition.