SpecPL: Disentangling Spectral Granularity for Prompt Learning

Existing prompt learning for VLMs exhibits a modality asymmetry, predominantly optimizing text tokens while still relying on frozen visual encoder as holistic extractor and neglecting the spectral granularity essential for fine-grained discrimination. To bridge this, we introduce Disentangling Spectral Granularity for Prompt Learning (SpecPL), which approaches prompt learning from a novel spectral perspective via Counterfactual Granule Supervision. Specifically, we leverage a frozen VAE to decompose visual signals into semantic low-frequency bands and granular high-frequency details. A frozen Visual Semantic Bank anchors text representations to universal low-frequency invariants, mitigating overfitting. Crucially, fine-grained discrimination is driven by counterfactual granule training: by permuting high-frequency signals, we compel the model to explicitly distinguish visual granularity from semantic invariance. Uniquely, SpecPL serves as a universal plug-and-play booster, revitalizing text-oriented baselines like CoOp and MaPLe via visual-side guidance. Experiments on 11 benchmarks demonstrate competitive state-of-the-art performance, achieving a new performance ceiling of 81.51\% harmonic-mean accuracy. These results validate that spectral disentanglement with counterfactual supervision effectively bridges the gap in the stability-generalization trade-off.

Pre-trained AI models that understand both images and text are powerful, but adapting them to specific new tasks efficiently is a core challenge. Current adaptation methods typically focus on updating the text instructions while treating the image as a fixed, unchangeable block. This causes the AI to miss crucial fine details and struggle to generalize when faced with new image categories or scarce training data. To address this, we developed SpecPL, which splits visual information into two distinct parts: a stable "base" (capturing overall shape and structure) and fine "details" (capturing specific textures). By separating these, we teach the AI to anchor its understanding to broad, reliable concepts, while simultaneously forcing it to pay close attention to the subtle visual clues that distinguish specific objects. SpecPL acts as an easy-to-add booster for existing AI models, significantly improving their ability to recognize unseen categories and handle shifts in image styles. This leads to more reliable AI systems for complex real-world tasks without slowing down the model during actual use.