ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate object labels and geometry across 413 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D visualization and annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.

Vision-language models are increasingly expected to answer questions about videos that require understanding 3D spaces, such as how many chairs are in a room or how far one object is from another. However, evaluating this ability is harder than it may seem: in prior work such as VSI-Bench, some questions rely on incorrect scene information or ask about objects that are not actually visible to the model. In this work, we rebuild VSI-Bench into a more reliable benchmark for visual spatial intelligence, ensuring that each question is supported by the visual evidence available to the model. Our benchmark, ReVSI, corrects object and room annotations across 381 real indoor scenes, regenerates questions with human verification, and provides evaluation settings for different numbers of video frames. We also design controlled tests that remove task-relevant visual evidence to examine whether models truly use the video or instead rely on guesses from common indoor scenes. ReVSI shows that some previous conclusions about model performance were misleading. More broadly, our work provides a more reliable way to measure whether vision-language models can reason about 3D spaces, which is important for future applications in robotics, video understanding, and embodied AI.