Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining

Heterogeneous multi-modal remote sensing object detection aims to accurately detect objects from diverse sensors (e.g., RGB, SAR, Infrared). Existing approaches largely adopt a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning. This tight coupling complicates optimization and often results in unstable training and suboptimal generalization. To address these limitations, we propose BabelRS, a unified language-pivoted pretraining framework that explicitly decouples modality alignment from downstream task learning. BabelRS comprises two key components: Concept-Shared Instruction Aligning (CSIA) and Layerwise Visual-Semantic Annealing (LVSA). CSIA aligns each sensor modality to a shared set of linguistic concepts, using language as a semantic pivot to bridge heterogeneous visual representations. To further mitigate the granularity mismatch between high-level language representations and dense detection objectives, LVSA progressively aggregates multi-scale visual features to provide fine-grained semantic guidance. Extensive experiments demonstrate that BabelRS stabilizes training and consistently outperforms state-of-the-art methods without bells and whistles. Code: \href{https://github.com/zcablii/SM3Det}{github.com/zcablii/SM3Det}.

Modern remote sensing systems collect images from very different sensors, such as optical cameras, radar (SAR), and infrared devices. While combining these sources could improve object detection (e.g., identifying vehicles or buildings), existing methods struggle because they try to align these very different data types while simultaneously learning the detection task. This often leads to unstable training and poor generalization, especially when the data are not perfectly aligned across sensors. In this work, we propose BabelRS, a new framework that separates these two challenges. Instead of aligning modalities during task training, we first align them during a dedicated pretraining stage using language as a common semantic bridge. Specifically, we train the model so that images from different sensors describing the same concept (e.g., “car”) produce consistent language-based representations. We further introduce a mechanism that gradually incorporates fine-grained visual details, making the model suitable for precise object detection. Our approach leads to more stable training and consistently better performance across multiple sensor types, without requiring paired multi-modal data. This makes it more practical for real-world remote sensing applications, where such data are often unavailable, and provides a scalable path toward unified multi-modal Earth observation systems.