UrbanMLLM: Joint Learning of Cross-view Imagery for Urban Understanding

Comprehensive urban understanding requires integrating macroscopic spatial structure with fine-grained street-level semantics. However, existing urban Multimodal Large Language Models (MLLMs) primarily rely on satellite imagery, limiting their ability to capture detailed urban appearance and cross-view relationships. We propose \textbf{UrbanMLLM}, a unified MLLM that jointly learns from satellite and street-view imagery for cross-view urban perception and reasoning. To support this, we construct a large-scale dataset with paired cross-view urban images, geospatial alignment, and textual annotations. UrbanMLLM introduces a cross-view perceiver to explicitly model interactions between satellite and street-view representations, and adopts a structural interleaved pre-training paradigm that organizes cross-view image–text content as coherent urban documents to enhance cross-view knowledge fusion. We evaluate UrbanMLLM on 13 diverse urban understanding tasks spanning satellite, street-view, and cross-view settings. Experimental results demonstrate consistent improvements over strong open-source and proprietary MLLMs, highlighting effectiveness and scalability of UrbanMLLM for urban environment understanding.

Many important questions about cities, such as how neighborhoods are built, how streets look, and how people experience urban spaces, cannot be answered well from satellite images alone. From above, we can see the overall layout of a city, but we miss many details visible at street level, such as building appearance, road conditions, and the character of public spaces. In this work, we build a new AI system called UrbanMLLM that learns from both satellite images and street-view photos at the same time. We also created a large dataset that links these two views of the same urban locations together with written descriptions. This helps the system connect the big picture of a city with the details people see on the ground. We tested our method on 13 different urban understanding tasks and found that it performed better than strong existing systems. Our results show that combining these two views can lead to more accurate and useful tools for understanding urban environments.