Solving Spatial-Spectral Fusion with Latent Spectral Operators

Existing deep spatial–spectral fusion (SSF) methods typically learn the fusion mapping in the coordinate domain using convolutions and attentions, making it hard to scale across varying spatial resolutions and offering limited control over the frequency content of the reconstructions, which may further lead to severe spectral distortion. In this work, we propose Latent Spectral Operators (LSO), a SSF framework that learns fusion mappings between spectral functions through a structured operator parameterization. Specifically, LSO first applies a cross-attention projection, where learned latent tokens serve as spectral prompts, to compress high-dimensional observations into a compact latent representation, and then adopts a hierarchical, patch-based architecture to integrate rich multi-scale cues. Furthermore, to parameterize the latent fusion operator in a controllable manner, a Trigonometric Basis Solver is elaborated, which represents the mapping using a trigonometric basis expansion. This formulation naturally supports multi-frequency modeling, with a capacity-stability trade-off governed by the number of basis functions. Extensive experiments on the CAVE and Harvard benchmarks demonstrate that LSO achieves consistent state-of-the-art performance and exhibits strong transferability across different spatial scales.

Hyperspectral images capture much richer information than ordinary color images because they record many narrow color bands instead of only red, green, and blue. This makes them useful for tasks such as environmental monitoring, disaster assessment, and remote sensing, but there is a practical trade-off: images with rich spectral information often have low spatial resolution, while sharp images usually contain fewer spectral bands. Our work studies how to combine these two types of observations to recover an image that is both spatially sharp and spectrally rich. Existing deep learning methods often process images directly in pixel space, which can make them less reliable when image resolutions change and may introduce unwanted spectral distortions. We propose Latent Spectral Operators, a method that first compresses the input images into a compact hidden representation and then learns the fusion process using controllable wave-like basis functions. This design helps the model preserve smooth and accurate spectral information while recovering fine spatial details. Experiments on standard benchmarks show that our method achieves strong reconstruction quality and generalizes well across different image scales.