Rethinking Federated Prompt Learning for Medical Images: From Textual Tuning to Visual Manifold Anchoring

Federated Prompt Learning (FPL) adapts Vision-Language Models to privacy-sensitive medical imaging, typically via a textual tuning paradigm that assumes the frozen visual encoder provides a discriminative feature geometry. We argue this assumption breaks down in medical settings, leading to two geometric pathologies: (1) Intra-client: Medical Manifold Collapse, where high morphological similarity reduces the effective rank of visual features; and (2) Inter-client: Medical Topological Misalignment, where heterogeneous acquisition protocols induce inconsistent geometry across clients. To address these, we propose FedMAP, which shifts the paradigm to Visual Manifold Anchoring. FedMAP utilizes an LLM-derived codebook as a client-invariant synchronization signal to restructure the visual space, via Manifold Semantic Anchoring (MSA) and Topology Structural Alignment (TSA) to enforce consistent inter-class relations. Experiments on FedISIC, FedCamelyon17, and a private ultrasound dataset show that FedMAP consistently outperforms state-of-the-art methods, especially in high-noise regimes where manifold collapse is most severe.

Modern medical AI systems can learn from large collections of images and text, but hospitals often cannot share patient data. In this work, we study how to adapt such systems across hospitals without moving patient images. We find that a common approach, which adjusts only the text side of the model, can fail because medical images from different diseases may look very similar and because hospitals use different scanners or imaging protocols. These differences make the model organize images inconsistently at each site. We propose FedMAP, which gives every hospital a shared set of medically meaningful descriptions and uses them to guide how the image model represents local data. Only small model updates are exchanged, while patient data stay at each hospital. Across skin, tissue-slide, and ultrasound datasets, FedMAP makes the model more accurate and more stable across sites. We hope this helps the community build more reliable privacy-preserving medical AI for multi-center collaboration.