Beyond Description: Federated Adaptation via Semantic-Visual Prototype Alignment

Adopting pre-trained Vision-Language Models (VLMs) in Federated Learning (FL) presents a promising avenue for mitigating data scarcity and heterogeneity. However, existing solutions suffer from high computational complexity or ineffective knowledge aggregation. To address these problems, we propose FedSPA (Federated Adaptation via Semantic-Visual Prototype Alignment). On the client side, FedSPA restricts local optimization to visual prototypes, enabling lightweight personalization. On the server side, we introduce a semantic alignment module that leverages client-uploaded prototypes to minimize a contrastive objective, aligning global semantic prototypes with heterogeneous visual distributions and thereby shifting the paradigm from traditional "learning-to-describe" (optimizing static prompts) to "learning-to-align". Extensive experiments demonstrate that FedSPA significantly outperforms state-of-the-art methods in both personalized and global benchmarks, while substantially reducing computational overhead. The code is available at https://github.com/eejiarong/FedSPA-main.

Many real-world organizations, such as hospitals or mobile users, need to train image recognition models without sharing their private data. Federated learning makes this possible, but each client often has different data, and many existing methods are either too costly to run or fail to combine knowledge effectively. We propose FedSPA, a lightweight method that adopts a pre-trained vision-language model for federated learning. Each client only learns a small set of visual representatives from its local data, while the server learns shared semantic representatives by aligning them with client-uploaded visual information. This design helps the global model capture common knowledge while preserving local personalization. Experiments show that FedSPA improves accuracy and greatly reduces computation compared with existing methods.