HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion

Remote sensing pansharpening aims to reconstruct spatial-spectral properties during the fusion of panchromatic (PAN) images and low-resolution multi-spectral (LR-MS) images, finally generating the high-resolution multi-spectral (HR-MS) images. In the mainstream modeling strategies, i.e., CNN and Transformer, the input images are treated as the equal-sized grid of pixels in the Euclidean space. They have limitations in facing remote sensing images with irregular ground objects. Graph is the more flexible structure, however, there are two major challenges when modeling spatial-spectral properties with graph: 1) constructing the customized graph structure for spatial-spectral relationship priors; 2) learning the unified spatial-spectral representation through the graph. To address these challenges, we propose the spatial-spectral heterogeneous graph learning network, named HetSSNet.Specifically, HetSSNet initially constructs the heterogeneous graph structure for pansharpening, which explicitly describes pansharpening-specific relationships. Subsequently, the basic relationship pattern generation module is designed to extract the multiple relationship patterns from the heterogeneous graph. Finally, relationship pattern aggregation module is exploited to collaborativelylearn unified spatial-spectral representation across different relationships among nodes with adaptive importance learning from local and global perspectives. Extensive experiments demonstrate the significant superiority and generalization of HetSSNet.

Remote sensing pansharpening aims to fuse high-resolution panchromatic (PAN) images with low-resolution multispectral (LR-MS) images to reconstruct spatial-spectral information and generate high-resolution multispectral (HR-MS) images. While mainstream approaches like CNNs and Transformers process images as regular pixel grids in Euclidean space, they struggle with irregular ground objects in real-world remote sensing scenes. Graph-based methods offer greater flexibility, but two key challenges remain: (1) designing a graph structure that captures spatial-spectral relationships, and (2) learning a unified representation from such a graph.To address these challenges, we propose HetSSNet, a spatial-spectral heterogeneous graph learning network. HetSSNet first constructs a task-specific heterogeneous graph to model pansharpening relationships explicitly. Next, a relationship pattern generation module extracts multiple interaction patterns from the graph. Finally, a relationship pattern aggregation module adaptively fuses these patterns, learning a unified spatial-spectral representation by combining local and global perspectives.Extensive experiments demonstrate HetSSNet’s superior performance and strong generalization capability compared to existing methods.