Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial reasoning. In this work, we analyze the geospatial representations acquired by three model families: vision-only architectures (e.g., ViT), vision-language models (e.g., CLIP), and large-scale multimodal foundation models (e.g., LLaVA, Qwen, and Gemma). By evaluating across image clusters, including people, landmarks, and everyday objects, grouped based on the degree of localizability, we reveal systematic gaps in spatial accuracy and show that textual supervision enhances the learning of geospatial representations. Our findings suggest the role of language as an effective complementary modality for encoding spatial context and multimodal learning as a key direction for advancing geospatial AI.

Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial reasoning. In this work, we explore how representations of geospatial information appear in vision-only and vision-language models by prompting the models with images from different semantic groups (e.g., people, objects, landmarks). By examining how models represent images at each layer and mapping these representations to real-world coordinates, we identify the common factors that lead to higher spatial accuracy. We find that models trained with textual supervision in addition to images have enhanced geospatial representations. Our findings suggest that language helps models to encode spatial context, and that multimodal learning is a key direction for advancing geospatial AI.