Benchmarking the Scientific Mind: A Pathology-Derived Biomedical VQA Benchmark for Complex Scientific Reasoning

Despite progress of Multimodal Large Language Models (MLLMs) in biomedical visual question answering (VQA), existing benchmarks provide limited assessment of their scientific reasoning capabilities. Most datasets adopt single-image question construction and outcome-oriented evaluation, where correctness is judged by answer plausibility rather than alignment with experimental evidence. Such formulations fail to capture the evidence-constrained, multi-step nature of biomedical reasoning, and obscure whether models can derive conclusions through causal interpretation of experimental observations. To address these critical gaps in reasoning evaluation, we propose a principled benchmark construction framework that reconstructs scientific reasoning paths directly from biomedical literature. By jointly modeling clusters of experimentally related images together with their captions and context, the framework generates tightly coupled question–reasoning–answer triples that require multi-image integration and explicit evidence-driven inference. Based on this framework, we introduce SORBE (Scientific Observation & Reasoning for Biomedical Evaluation), a large-scale multi-image pathology-derived biomedical VQA benchmark designed to evaluate evidence alignment and multi-step experimental reasoning. Under a process-oriented evaluation metric, state-of-the-art biomedical-specialized MLLMs exhibit substantial performance degradation, revealing systematic limitations in evidence grounding and causal reasoning that are not reflected by existing benchmarks. Data and code are available at: https://github.com/UniverseOfUniverse/SORBE.git.

AI systems are becoming better at answering questions about biomedical images, but current tests often only check whether a system gives a reasonable final answer. This can be misleading, because a model may sound correct without actually using the visual evidence in the image or following the scientific logic behind an experiment. This paper introduces a new way to test whether AI models can reason more like scientists when working with biomedical research images. Instead of asking questions about a single image in isolation, we build questions from groups of related images, their captions, and the surrounding scientific context. These questions require the model to connect several pieces of evidence and explain how they support a conclusion. Based on this idea, we create SORBE, a new biomedical image question-answering benchmark focused on evidence-based scientific reasoning. SORBE evaluates not only whether a model gives the right final answer, but also whether its reasoning process follows the evidence step by step. Our experiments show that even strong current biomedical AI models perform much worse on SORBE than on standard biomedical question-answering tests. This suggests that today’s models still have important limitations in grounding their answers in evidence and making reliable scientific judgments.