VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

Current multimodal benchmarks often conflate reasoning with domain knowledge, making it difficult to isolate and evaluate reasoning abilities in non-expert settings. To address this, we introduce VisualPuzzles, a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of diverse puzzles spanning 5 categories: algorithmic, analogical, deductive, inductive, and spatial reasoning. One major source of our questions is manually translated logical reasoning questions from the Chinese Civil Service Examination. Experiments show that VisualPuzzles requires significantly less intensive domain-specific knowledge and more complex reasoning compared to benchmarks like MMMU, enabling us to better evaluate genuine multimodal reasoning. Evaluations show that state-of-the-art multimodal large language models consistently lag behind human performance on VisualPuzzles, and that strong performance on knowledge-intensive benchmarks does not necessarily translate to success on reasoning-focused, knowledge-light tasks. Additionally, reasoning enhancements such as scaling up inference compute (with "thinking" modes) yield inconsistent gains across models and task types, and we observe no clear correlation between model size and performance. We also found that models exhibit different reasoning and answering patterns on VisualPuzzles compared to benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer lens through which to evaluate reasoning capabilities beyond factual recall and domain knowledge.

Artificial intelligence systems are increasingly expected to understand images and answer questions about them. However, many existing tests mix two different abilities: knowing specialized facts and actually reasoning through a problem. This makes it hard to tell whether an AI model is truly reasoning or simply recalling learned knowledge. In this paper, we introduce VisualPuzzles, a new test designed to better measure visual reasoning. The questions are puzzle-like and require only the information shown in the image, the question text, and basic common sense, rather than expert knowledge. The puzzles cover different kinds of reasoning, such as finding patterns, making logical deductions, understanding spatial relationships, and comparing visual structures. We evaluate many current AI models and compare them with human performance. We find that even strong multimodal models still perform far below humans on these puzzles. We also find that models that do well on knowledge-heavy benchmarks do not necessarily do well on VisualPuzzles, suggesting that visual reasoning remains a distinct and challenging problem. Our benchmark provides a clearer way to study and improve AI systems’ ability to reason about visual information.