MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

Frontier models are transitioning from multimodal large language models (MLLMs) that merely ingest visual information to unified multimodal models (UMMs) capable of native interleaved generation. This shift has sparked interest in using intermediate visualizations as a reasoning aid, akin to human mental imagery. Central to this idea is the ability to form, maintain, and manipulate visual representations in a goal-oriented manner. To evaluate and probe this capability, we develop MentisOculi, a procedural, stratified suite of multi-step reasoning problems amenable to visual solution, tuned to challenge frontier models. Evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance. Analysis of UMMs specifically exposes a critical limitation: While they possess the textual reasoning capacity to solve a task and can sometimes generate correct visuals, they suffer from compounding generation errors and fail to leverage even ground-truth visualizations. Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning. MentisOculi establishes the necessary foundation to analyze and close this gap across diverse model families.

Humans often solve complex spatial problems, like figuring out how to pack a crowded car trunk, by visualizing the steps in our minds. As artificial intelligence models become capable of generating both text and pictures, we wonder if AI could also use this kind of "mental imagery" to solve tricky spatial problems. However, it is unclear if creating images actually improves an AI's ability to reason. To test this, we built a new collection of visual puzzles called MentisOculi. These include tasks like sliding car games and paper-folding challenges that require step-by-step visual thinking. We tested the most advanced AI models to see if generating intermediate pictures helped them find the right answers. Surprisingly, we found that "thinking with pictures" does not currently help these models. Even when the AI successfully draws the correct intermediate steps, it fails to actually understand and use those pictures to reach the final solution. Our results reveal a major blind spot in current AI systems—they can create images but cannot yet logically reason with them—and our puzzles provide a roadmap to help developers bridge this gap in the future.