Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation under insufficient local data while preserving generalization. However, these methods emphasize fully local optimization with simple parameter aggregation, which can amplify inter-client optimization inconsistency and intra-client over-specialization under heterogeneous and full-data FL settings, making it difficult to balance global task adaptation and generalization. To address these challenges, we propose FedDTL, a novel federated VLM framework that decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, FedDTL promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation. To further mitigate intra-client over-specialization, we introduce a two-stage local fine-tuning, where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Extensive experiments on multiple benchmarks, including label skew and feature shift, demonstrate that FedDTL achieves an effective balance between global task adaptation and generalization under various FL data distributions in both few-shot and full-data regimes.

Federated learning enables multiple clients to collaboratively train machine learning models (such as AI models) without sharing sensitive raw data. However, clients often possess local data with diverse distributions. Such heterogeneous data distributions make it difficult for models to learn knowledge and perform well across all clients, degrading the global model performance. We propose FedDTL to address this challenge. FedDTL improves global task adaptation through a decoupled training scheme based on pre-trained vision-language models between clients and a server. Moreover, the proposed two-stage local fine-tuning mechanism in FedDTL further enhances the model’s generalization ability. Experimental results demonstrate that FedDTL can achieve stronger model performance across diverse data settings. Therefore, our findings facilitate the development of more secure and efficient federated learning with AI systems, supporting potential applications in real-world scenarios such as healthcare and intelligent devices.