Imagination Helps Visual Reasoning, But Not Yet in Latent Space

Latent visual reasoning aims to mimic human's imagination process by meditating through hidden states of Multimodal Large Language Models. While recognized as a promising paradigm for visual reasoning, the underlying mechanisms driving its effectiveness remain unclear. Motivated to demystify the true source of its efficacy, we investigate the validity of latent reasoning using Causal Mediation Analysis. We model the process as a causal chain: the input as the treatment, the latent tokens as the mediator, and the final answer as the outcome. Our findings uncover two critical disconnections: (a) Input-Latent Disconnect: dramatic perturbations on the input result in negligible changes to the latent tokens, suggesting that latent tokens do not effectively attend to the input sequence. (b) Latent-Answer Disconnect: perturbations on the latent tokens yield minimal impact on the final answer, indicating the limited causal effect latent tokens imposing on the outcome. Furthermore, extensive probing analysis reveals that latent tokens encode limited visual information and exhibit high similarity. Consequently, we challenge the necessity of latent reasoning and propose a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text. Experiments on vision-centric benchmarks show that CapImagine significantly outperforms complex latent-space baselines, highlighting the superior potential of visual reasoning through explicit imagination.

How do advanced AI models "imagine" when solving complex visual problems? Recently, researchers have tried teaching them to "think" using hidden, internal steps (technically called latent tokens) before giving an answer. We wanted to understand if these hidden steps actually drive the model's success. By mathematically tracing cause and effect within the model, we uncovered a surprising result: these hidden steps are largely ignored. We found that drastically changing the input question barely affects the hidden "thoughts," and altering those thoughts barely changes the final answer. Furthermore, these internal states hold very little actual visual information. Essentially, the model is not really using them to reason. Because these hidden steps proved unnecessary, we created a much simpler approach called CapImagine for analysis and comparison. Instead of relying on unreadable internal states, our method teaches the AI to explicitly "imagine" by describing what it sees in plain text. When tested on challenging visual tasks, this straightforward text-based imagination significantly outperformed the complex hidden-step methods, revealing the unrealized potential for current latent-based methods.