Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions

Multimodal Large Language Models (MLLMs) demonstrate impressive cross-modal capabilities, yet their substantial size poses significant deployment challenges. Knowledge distillation (KD) is a promising solution for compressing these models, but existing methods primarily rely on static next-token alignment, neglecting the dynamic token interactions, which embed essential capabilities for multimodal understanding and generation. To this end, we introduce Align-TI, a novel KD framework designed from the perspective of Token Interactions. Our approach is motivated by the insight that MLLMs rely on two primary interactions: vision-instruction token interactions to extract relevant visual information, and intra-response token interactions for coherent generation. Accordingly, Align-TI introduces two components: IVA enables the student model to imitate the teacher's instruction-relevant visual information extract capability by aligning on salient visual regions. TPA captures the teacher's dynamic generative logic by aligning the sequential token-to-token transition probabilities. Extensive experiments demonstrate Align-TI's superiority. Notably, our approach achieves 2.6% relative improvement over Vanilla KD, and our distilled Align-TI-2B even outperforms LLaVA-1.5-7B (a much larger MLLM) by 7.0%, establishing a new state-of-the-art distillation framework for training parameter-efficient MLLMs.

Multimodal large language models can understand both images and text, but strong models are usually very large and expensive to deploy. This paper studies how to train smaller multimodal models by transferring knowledge from larger teacher models more effectively. Instead of only asking the small model to copy the teacher’s next-token predictions, our method, Align-TI, teaches the student model to learn how the teacher focuses on instruction-relevant visual regions and how it maintains coherent token-to-token generation behavior. These two forms of token interaction help the smaller model better understand visual inputs and generate more reliable responses. Experiments show that Align-TI improves compact multimodal models and can even outperform some larger baselines, making capable multimodal AI more efficient and accessible.