SPR: A Structured Prompt Refinement Network for Modality Missing

Prompt learning has recently emerged as a dominant paradigm to tackle the missing modalities challenge. However, existing methods often overlook the internal structural information of prompt vectors, limiting performance in guiding frozen backbone models under diverse missing modality scenarios. To address this, we propose a Structured Prompt Refining (SPR) network that refines the internal structure of prompt vectors across multiple dimensions: (1) a Global Interaction Fusion Module captures bidirectional interactions across prompt layers, thereby mitigating sub-optimal adaptation from inconsistent guidance under missing modalities, (2) a Local Feature Refinement Module structures adjacent prompt vectors into coherent semantic units, leveraging local contextual relationships to maintain semantic integrity during modality absence, and (3) a Channel Feature Selection Module uses point-wise gating to adaptively suppress noise and enhance critical channels based on the specific missing modality. Using only 0.8% trainable parameters, SPR achieves significant improvements on three mainstream multimodal classification datasets. Notably, it surpasses state-of-the-art by 3.8% in F1-Macro on the MM-IMDB dataset, even at a 90% modality missing rate. Extensive experiments and in-depth ablations validate SPR's effectiveness and robustness under various missing conditions.

To address the challenge of missing modalities in multimodal learning, this paper proposes a highly efficient Structured Prompt Refinement (SPR) network. In contrast to existing methods that fail to fully exploit the internal structural information of prompt vectors, SPR multi-dimensionally optimizes prompts through three specialized modules: the Global Interaction Fusion Module, the Local Feature Refinement Module, and the Channel Feature Selection Module, thereby actively reconstructing missing semantic information. With this innovative approach, SPR introduces a mere 0.8% of trainable parameters yet significantly outperforms state-of-the-art methods, achieving a remarkable F1-Macro score of 53.41% even under an extreme 90% modality missing rate on the MM-IMDb dataset. Consequently, it effectively prevents representational collapse while delivering exceptional robustness and generalization performance.