UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data. Current methods primarily utilize task-specific models, while recent generic models for spatial representations often support only limited modalities and lack multimodal fusion capabilities. To overcome these challenges, we present UrbanFusion, a spatial representation model that features Stochastic Multimodal Fusion (SMF). The framework employs modality-specific encoders to process different types of inputs, including street view imagery, remote sensing data, cartographic maps, and points of interest (POIs) data. These multimodal inputs are integrated via a Transformer-based fusion module that learns unified representations. An extensive evaluation across 41 tasks in 56 cities worldwide demonstrates UrbanFusion’s strong generalization and predictive performance compared to state-of-the-art GeoAI models. Specifically, it 1) outperforms prior models on location-encoding, 2) allows multimodal input during inference, and 3) generalizes well to regions unseen during training. UrbanFusion can flexibly utilize any subset of available modalities for a given location during both pretraining and inference, enabling broad applicability across diverse data availability scenarios.

Cities are complex, and predicting outcomes such as housing prices, energy use, public health, crime, or land use often requires combining many kinds of geographic information. Existing geospatial AI models usually rely on only one or two data sources and can struggle when data is incomplete or when used in new cities. We introduce UrbanFusion, a model that represents urban locations using coordinates, street-view images, satellite imagery, maps, and nearby points of interest. Its key innovation is Stochastic Multimodal Fusion, a learning framework that randomly splits available data sources into subsets, aligns these partial views of the same place, and reconstructs missing information. This helps the model capture shared, source-specific, and combined information across modalities. Across many urban prediction tasks worldwide, UrbanFusion generally outperforms existing geospatial AI models and remains useful when some data sources are missing.