TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens ($\langle\text{Think}\rangle$, $\langle\text{Look}\rangle$, $\langle\text{Answer}\rangle$). These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1\% on MMMU, +3.8\% on MathVerse, +3.4\% on MathVista, and +3.4\% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Today’s AI systems that understand both images and text can solve problems step-by-step, but they have a big flaw: once they “look” at an image at the very start, they can’t look back during reasoning—they just think blindly using text. This makes them bad at complex visual tasks like geometry or science problems that need checking details multiple times. This paper introduces TVI-CoT, a new way to make AI reason more like humans: it uses special simple commands, i.e.,Think, Look, Answer, to switch back and forth between thinking in text and checking the image as needed. When stuck, the AI pauses to “Look” at the right part of the image, gathers new visual clues, and then keeps “Thinking” until it’s ready to “Answer.” Tests show this method makes AI much better at tricky image-based questions, especially math and science problems with diagrams, while staying efficient and easy to understand.