From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models

While multimodal large language models (MLLMs) have made substantial progress in single-image spatial reasoning, multi-image spatial reasoning, which requires integration of information from multiple viewpoints, remains challenging. Cognitive studies suggest that humans address such tasks through two mechanisms: cross-view correspondence, which identifies regions across different views that correspond to the same physical locations, and stepwise viewpoint transformation, which composes relative viewpoint changes sequentially. However, existing studies incorporate these mechanisms only partially and often implicitly, without explicit supervision for both. We propose Human-Aware Training for Cross-view correspondence and viewpoint cHange (HATCH), a training framework with two complementary objectives: (1) Patch-Level Spatial Alignment, which encourages patch representations to align across views for spatially corresponding regions, and (2) Action-then-Answer Reasoning, which requires the model to generate explicit viewpoint transition actions before predicting the final answer. Experiments on three benchmarks demonstrate that HATCH consistently outperforms baselines of comparable size by a clear margin and achieves competitive results against much larger models, while preserving single-image reasoning capabilities.

When you walk into a room and glance around, you can piece together where everything is, such as the desk behind you or the window to your left, even though you never see the whole room in a single glance. AI systems that understand images struggle with this. Given several photos of the same place taken from different angles, they often fail to figure out how the views fit together. This limits their usefulness for things like robots navigating a building or systems analyzing footage from multiple cameras. We took inspiration from how people solve this. Humans do two things. They recognize that an object in one photo is the same object seen in another, and they mentally "walk" from one viewpoint to the next, step by step. We built a training method, called HATCH, that explicitly teaches an AI both of these skills: matching the same locations across photos, and describing the camera movements needed to get from one view to another before answering. Despite using a relatively small AI model, our approach outperforms much larger ones at reasoning about space across multiple images, while still handling single images well. This brings AI a step closer to understanding physical spaces the way humans naturally do.