Rethinking Federated Prompt Learning for Medical Images: From Textual Tuning to Visual Manifold Anchoring

Federated Prompt Learning (FPL) adapts Vision-Language Models to privacy-sensitive medical imaging, typically via a textual tuning paradigm that assumes the frozen visual encoder provides a discriminative feature geometry. We argue this assumption breaks down in medical settings, leading to two geometric pathologies: (1) Intra-client: Medical Manifold Collapse, where high morphological similarity reduces the effective rank of visual features; and (2) Inter-client: Medical Topological Misalignment, where heterogeneous acquisition protocols induce inconsistent geometry across clients. To address these, we propose FedMAP, which shifts the paradigm to Visual Manifold Anchoring. FedMAP utilizes an LLM-derived codebook as a client-invariant synchronization signal to restructure the visual space, via Manifold Semantic Anchoring (MSA) and Topology Structural Alignment (TSA) to enforce consistent inter-class relations. Experiments on FedISIC, FedCamelyon17, and a private ultrasound dataset show that FedMAP consistently outperforms state-of-the-art methods, especially in high-noise regimes where manifold collapse is most severe.