LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM. Intriguingly, this mapping can be as simple as a shallow MLP transformation. To understand why LLMs can so readily process visual tokens, we need interpretability methods that reveal what is encoded in the visual token representations at *every* layer of LLM processing. In this work, we introduce LatentLens, a novel approach for mapping latent representations to descriptions in natural language. LatentLens encodes a large text corpus and stores contextualized token representations for each token in that corpus. Visual token representations are then compared to these contextualized representations and the top-$k$ nearest neighbor representations serve as descriptions of the visual token. We evaluate this method on 15 different VLMs, showing that commonly used methods, such as LogitLens, substantially underestimate the interpretability of visual tokens. With LatentLens instead, the majority of visual tokens are interpretable across all studied models and all layers. Qualitatively, we show that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens. More broadly, our findings contribute new evidence on the alignment between vision and language representations and open up new directions for analyzing the latent representations of LLMs.

Large Language Models (LLMs) are now used everywhere with remarkable success and yet they are trained in a very simple way: predicting the next word given the previous words. With this training they learn a lot about the world in general. And despite being trained to only process text input, somehow they can very quickly be adapted to also understand the visual world (images, videos, ...). How do they accomplish being such good visual models as well? The field of "interpretability" tries to answer such questions: How do models internally represent knowledge, how do they arrive at certain decisions and generated content? In this specific study, we propose a new interpretability tool to understand whether the LLM translates "visual tokens", so small regions of the images, into something like "words". We hypothesized that the only way a model that knows the abstract world of language can make sense of images is by "translating" them into meaningful words. And our hypothesis proved correct: most visual tokens are indeed represented similar to the inherent word space of the LLM. With such a tool, which we call LatentLens, we can now better understand why an LLM might sometimes fail to answer questions about images or videos. And our findings also get us closer to a scientific understanding of the internal complex workings of AI models.