Towards Multimodal Large Language Models with Both Training and Inference Efficiency

Multimodal Large Language Models (MLLMs) mainly fall into two architectures, each involving a trade-off between training and inference efficiency: embedding space alignment (e.g. LLaVA series) is inefficient during inference, while cross-attention space alignment (e.g. Flamingo) is inefficient in training. A primary difference between them lies in whether each visual token attends to other tokens within the LLM backbones. To investigate whether this form of attention is essential for MLLMs, we propose NAEViT (No AttEntion from Visual Tokens), an attention mechanism that eliminates such interactions. Our pilot experiment shows that attention from visual tokens is highly redundant. Then, we introduce SAISA (Self-Attention Input Space Alignment), a novel architecture that enhances both training and inference efficiency. SAISA directly aligns visual features with the input spaces of NAEViT attention blocks, reducing computational overhead in both attention and FFNs. We conduct experiments on various baseline models, model sizes and training datasets. SAISA achieves superior performance compared to the baselines, while significantly reducing computational costs. Further ablation studies validate the effectiveness of SAISA across various LLMs and visual encoders.

Multimodal large language models are AI systems that can understand both images and text. Today, these models are often expensive to train or expensive to use, depending on how they connect visual information with language models. This cost makes it harder to build and deploy powerful multimodal AI systems in real-world applications. In this paper, we study whether all visual information really needs to interact with the language model in the usual way. We find that many of these interactions are unnecessary. Based on this observation, we design a new model architecture, called SAISA, that allows the model to process images and text more efficiently. SAISA reduces the amount of computation needed during both training and inference, while keeping or even improving model performance. We test SAISA across different model sizes, visual encoders, and training settings, and show that it consistently works well. Our results suggest that multimodal AI systems can be made faster and cheaper without sacrificing their ability to understand images and language.