Deep Pre-Alignment for VLMs

Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9\% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.

Many AI systems that understand both images and text connect an image reader directly to a language model. Although this works well, it can leave the language model spending part of its effort simply translating image information into a form it can use, rather than focusing on deeper understanding and reasoning. We propose Deep Pre-Alignment, a simple change to this design. Instead of using a conventional image reader, we use a small vision-language model to process images before passing the information to the larger language model. This makes the image information easier for the language model to understand from the start. Across a range of image-and-text tasks, our method improves performance for both smaller and larger models. It also helps preserve the model’s original text abilities, which can otherwise be weakened when adapting a language model to handle images. Importantly, the method is modular: existing vision-language systems can adopt it by replacing the image-processing component, with only a small increase in computation.