Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs

Recent Multimodal Large Language Models (MLLMs) have demonstrated significant progress in perceiving and reasoning over multimodal inquiries, ushering in a new research era for foundation models. However, vision-language misalignment in MLLMs has emerged as a critical challenge, where the textual responses generated by these models are not factually aligned with the given text-image inputs. Existing efforts to address vision-language misalignment have focused on developing specialized vision-language connectors or leveraging visual instruction tuning from diverse domains. In this paper, we tackle this issue from a fundamental yet unexplored perspective by revisiting the core architecture of MLLMs. Most MLLMs are typically built on decoder-only LLMs consisting of a causal attention mechanism, which limits the ability of the earlier modalities (e.g., images) to incorporate information from the latter modalities (e.g., text). To address this problem a MLLM that unlocks causal attention into our proposed modality-mutual attention (MMA) to enable image tokens to attend to text tokens. This simple yet effective design allows MMA to achieve state-of-the-art performance in 12 multimodal understanding benchmarks (+6.2\% on average across 3 LLMs backbones) without introducing additional parameters. Our MMA design is intended to be generic, allowing for applications across various modalities, and scalable to accommodate diverse multimodal scenarios.

Multimodal large language models (MLLMs) are designed to understand both images and text, enabling applications such as visual question answering and AI assistants. However, these models often generate responses that are inconsistent with the visual information they receive, leading to incorrect or misleading outputs. Existing approaches mainly address this problem by adding specialized modules or training on more multimodal data, while the underlying model architecture has remained largely unchanged. In this work, we identify a fundamental limitation in how current MLLMs process multimodal inputs. Most models use a causal attention mechanism that prevents earlier inputs, such as images, from effectively incorporating information from later inputs, such as text. To overcome this issue, we propose a new mechanism called modality-mutual attention (MMA), which allows image and text representations to interact with each other more directly. Despite its simplicity, MMA significantly improves multimodal understanding without adding extra model parameters. Across 12 multimodal benchmarks and three different model backbones, our method achieves state-of-the-art performance with an average improvement of 6.2%. Our findings show that revisiting the core architecture of MLLMs can substantially improve how AI systems align visual and textual information, paving the way for more reliable multimodal assistants.