Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding

Large vision–language models (LVLMs) perform well on multimodal tasks, but their ability to reason and precisely align visual and textual information still has room for improvement. In this study, we show that external visual cues, such as symbols or grid lines, help LVLMs form more accurate connections between visual components, such as objects, and their corresponding textual descriptions, improving their grounding and reasoning abilities. We introduce the concept of Grounding IDs, which are latent identifiers that arise within the model as a result of external cues structuring both visual and textual modalities. Our analysis reveals that partition-inducing external cues lead to Grounding IDs that make better alignment between corresponding visual and text representations, helping the model focus on relevant information. We find that Grounding IDs enhance attention between related components, improving cross-modal grounding and reducing hallucinations. Overall, our results show that Grounding IDs are a key mechanism that enables external cues to improve cross-modal alignment, reduce errors, and enhance the overall performance of LVLMs across a range of multimodal tasks.

Vision-language models are AI systems that answer questions about images and generate descriptions of visual scenes. Although simple cues such as symbols, lines, or grids can improve their performance, it is not clear why these cues help or what changes inside the model when they are used. In this work, we study the internal mechanism behind this effect. We find that when matching cues are added to both the image and the text prompt, the model forms internal tags that help connect each visual region with its corresponding text. We call these tags Grounding IDs. Our main goal is to understand and explain this mechanism, rather than only measure performance improvements. Through a set of experiments, we show that these internal tags affect where the model pays attention and which object it chooses in its answer. By directly changing some internal representations of the model, we find evidence that these tags play an active role in linking objects to their descriptions. Overall, this work provides an interpretability-based explanation of how simple external cues improve visual grounding in vision-language models, and why they can reduce mistakes such as confusing objects or describing things that are not present.