Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out-of-distribution (OOD), while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model’s internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance and other encoder-level inductive biases, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes and visual encoders, and show that analogous shifts occur during pre-trained LLM-to-VLM transitions. Our findings suggest that cross‑modal training can enhance reasoning and generalization even for tasks grounded in a single modality.

Large Language Models are highly capable at processing text, but we have observed a surprising trend: when these models are upgraded to process images, their performance on purely text-based tasks actually improves. Specifically, these upgraded Vision-Language Models become better at retrieving information. This raises a puzzling question: why would teaching an artificial intelligence to see make it better at reading? To investigate this, we built a controlled test that required models to track colored shapes and retrieve information. We discovered that text-only models often rely on brittle shortcuts, basing their answers on where words are positioned in a sentence rather than linking their actual meanings. Because objects in images can appear anywhere on a canvas without a fixed order, adding visual training completely disrupts these positional shortcuts. This forces the artificial intelligence to learn a more robust symbolic strategy that correctly links concepts based on their semantic content, a stronger habit that it retains even when it switches back to processing text. Our research explains the hidden mechanics behind why Vision-Language Models consistently outperform their text-only counterparts in information retrieval. More importantly, it demonstrates that training artificial intelligence across multiple types of media is a powerful way to prevent models from learning fragile shortcuts, ultimately leading to more robust and reliable reasoning systems.