Multimodal Function Vectors for Visual Relations

Large Multimodal Models (LMMs) demonstrate impressive in-context learning abilities from few multimodal demonstrations, yet the internal mechanisms supporting such task learning remain opaque. Building on prior work of Large Language Models, we show that a small subset of attention heads in Large Multimodal Models is responsible for transmitting representations of visual relations. The activations of these attention heads, termed $\textit{function vectors}$, can be extracted and manipulated to alter an LMM’s performance on relational tasks. First, using synthetic and real image datasets, we apply causal mediation analysis to identify attention heads that strongly influence relational predictions, and extract multimodal function vectors that improve zero-shot accuracy at inference time. We further demonstrate that these multimodal function vectors can be fine-tuned with a modest amount of training data, while keeping LMM parameters frozen, to significantly outperform in-context learning baselines. Finally, we show that relation-specific function vectors can be linearly combined to solve analogy problems involving novel and untrained visual relations, highlighting the strong generalization ability of this approach. Through experiments on two LMMs, including OpenFlamingo and Qwen3-VL, our results show that these models encode visual relational knowledge within localized internal structures, which can be systematically extracted and optimized, thereby advancing our understanding of model modularity and enhancing control over relational reasoning in LMMs.

When we look at a kitchen, we don't just see a list of objects like "fridge, boy, cabinet." We see relationships: a boy opening the fridge, a cabinet next to a window. These relationships are what make a scene meaningful, but they aren't directly visible in any single pixel. AI systems that combine vision and language still struggle to recognize them reliably. We asked whether knowledge of visual relationships might already exist inside these models, hidden in a small part of their internal machinery. We found that roughly ten components out of thousands carry most of the information about a given relation, such as "above" or "holding." We can extract this as a compact numerical signature and inject it back into the model to substantially improve its accuracy, even without showing it any examples. These signatures can also be combined like building blocks: by mixing the signatures for "above" and "right," the model can recognize "above-right". This suggests vision-language models organize relational knowledge in a structured, modular way that researchers can read out, edit, and recombine.