VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference

Pursuing training-free open-vocabulary semantic segmentation in an efficient and generalizable manner remains challenging due to the deep-seated spatial bias in CLIP. To overcome the limitations of existing solutions, this work moves beyond the CLIP-based paradigm and harnesses the recent spatially-aware dino.txt framework to facilitate more efficient and high-quality dense prediction. While dino.txt exhibits robust spatial awareness, we find that the semantic ambiguity of text queries gives rise to severe mismatch within its dense cross-modal interactions. To address this, we introduce VIsual-guided Prompt evolution (VIP) to rectify the semantic expressiveness of text queries in dino.txt, unleashing its potential for fine-grained object perception. Towards this end, VIP integrates alias expansion with a visual-guided distillation mechanism to mine valuable semantic cues, which are robustly aggregated in a saliency-aware manner to yield a high-fidelity prediction. Extensive evaluations demonstrate that VIP: I) surpasses the top-leading methods by 1.4%~8.4\% average mIoU, II) generalizes well to diverse challenging domains, and III) requires marginal inference time and memory overhead.

Imagine asking a computer to color in every pixel of a photo, labeling the bus, the road, and the sky, even for categories it was never specifically trained on. This capability matters for self-driving cars, satellite analysis, and medical imaging. Existing methods either struggle to locate objects, or can locate them but fail to match them with plain category words such as "person." Our method, VIP, bridges this language gap. We prompt a large language model to generate multiple alternative phrases for each category (for example, "people" or "crowd" for "person"), then automatically filter out the unhelpful ones using visual cues from the image itself, without relying on any human labels, and combine the remaining phrases into a single accurate prediction. VIP achieves a new accuracy record on eight standard benchmarks, also performs well on aerial imagery, and runs up to fourteen times faster than leading alternatives.