Galileo: Learning Global & Local Features of Many Remote Sensing Modalities

We introduce a highly multimodal transformer to represent many remote sensing modalities - multispectral optical, synthetic aperture radar, elevation, weather, pseudo-labels, and more - across space and time. These inputs are useful for diverse remote sensing tasks, such as crop mapping and flood detection. However, learning shared representations of remote sensing data is challenging, given the diversity of relevant data modalities, and because objects of interest vary massively in scale, from small boats (1-2 pixels and fast) to glaciers (thousands of pixels and slow). We present a novel self-supervised learning algorithm that extracts multi-scale features across a flexible set of input modalities through masked modeling. Our dual global and local contrastive losses differ in their targets (deep representations vs. shallow input projections) and masking strategies (structured vs. not). Our Galileo is a single generalist model that outperforms SoTA specialist models for satellite images and pixel time series across eleven benchmarks and multiple tasks.

We capture a lot of information about our planet from “remote sensing data” (satellite observations, topographic maps, and more) but we know less than you might think. Analyzing remote sensing data with machine learning can help us better understand our changing planet.We present a machine learning model — which we call Galileo — that can help summarize remote sensing data. This means that with minimal further processing, its summaries can help make predictions and maps, like of floods or agricultural fields. We achieve this by giving Galileo an incomplete set of data for a time and place, and having it reconstruct what we removed. By being careful about exactly what we ask Galileo to reconstruct, we can make sure Galileo’s summaries take into account big and slow things (like glaciers) as well as small and fast things (like fishing boats).Galileo is uniquely relevant to remote sensing in practice by its modeling of data across space, time, and a variety of data types (e.g. optical data from satellites, topographic maps, weather data and more). We test Galileo on 15 diverse tasks against 11 other methods: Galileo performs best with a single general model. This makes it immediately useful in many existing applications.