Vision-aligned Latent Reasoning for Multi-modal Large Language Model

Despite recent advancements in Multi-modal Large Language Models (MLLMs) on diverse understanding tasks, these models struggle to solve problems which require extensive multi-step reasoning. This is primarily due to the progressive dilution of visual information during long-context generation, which hinders their ability to fully exploit test-time scaling. To address this issue, we introduce Vision-aligned Latent Reasoning (VaLR), a simple, yet effective reasoning framework that dynamically generates vision-aligned latent tokens before each Chain of Thought reasoning step, guiding the model to reason based on perceptual cues in the latent space. Specifically, VaLR is trained to preserve visual knowledge during reasoning by aligning intermediate embeddings of MLLM with those from vision encoders. Empirical results demonstrate that VaLR consistently outperforms existing approaches across a wide range of benchmarks requiring long-context understanding or precise visual perception, while exhibiting test-time scaling behavior not observed in prior MLLMs. In particular, VaLR improves the performance significantly from 33.0\% to 52.9\% on VSI-Bench, achieving a 19.9\%p gain over Qwen2.5-VL.

Multi-modal Large Language Models (MLLMs) excel at describing images and answering questions about them, yet they struggle with problems requiring many steps of reasoning. These models reason primarily in words, and as a chain of reasoning grows longer, the visual details from the original image gradually fade, leaving the model to reason from a hazy recollection rather than the image itself. We propose Vision-aligned Latent Reasoning (VaLR), a method that periodically refreshes the model's view of the image throughout reasoning. The key idea is to insert visual checkpoints before each reasoning step, prompting the model to re-anchor its thinking in what it actually perceived. During training, these checkpoints are guided to match the rich visual features produced by dedicated vision systems, enabling the model to stay grounded in perception even without additional visual aid at inference time. By preserving visual details, VaLR keeps improving when given more time to think, rather than drifting off course. More broadly, our work highlights the importance of perceptual reasoning for building MLLMs that can handle complex and vision-intensive tasks.