Heterogeneous Customizable Personalized Federated Fine-Tuning Approach for Large Language Models

Personalized federated LoRA fine tuning has become a key approach to addressing data heterogeneity in distributed fine tuning of large language models (LLMs). Existing methods typically assume homogeneous personalization needs across clients, relying on dual LoRA or personalized calibration schemes. However, they fail to account for the heterogeneity of local personalization requirements and the conflicting optimization objectives in dual LoRA, limiting scalability and performance. To address this, we propose Het-CPFLoRA, a customizable heterogeneous federated LoRA fine tuning algorithm inspired by the decoupling properties of LoRA parameters. We employ a single adapter fine tuning scheme to mitigate conflicts between personalized and generalized optimization, decouple LoRA into generalized and personalized subspaces for local customization, and use SVD compression to integrate cross client generalized knowledge. During inference, we introduce an OOD oriented dynamic mechanism to adjust the weighting between personalized and generalized decoupling knowledge, improving performance on user data. Extensive experiments on two public benchmark datasets show that Het-CPFLoRA outperforms state of the art methods in both personalization and generalization across heterogeneous scenarios. The code will be released as an open-source project.

Large language models are increasingly adapted using data stored by many users or organizations, without moving private data to a central server. However, different users often need different kinds of adaptation: for example, a medical assistant, a coding assistant, and a customer-service assistant may each require different knowledge. Existing federated fine-tuning methods usually treat these needs as similar, which can waste parameters and limit performance. We introduce Het-CPFLoRA, a method that lets each user customize how much of a lightweight model update is used for personal knowledge and how much is shared across users. Instead of training separate update modules that may conflict, our method splits one update module into shared and personalized parts, then efficiently combines shared knowledge across users. At test time, it also adjusts how much the model relies on shared versus personal knowledge for each input. Experiments on two public benchmarks show that Het-CPFLoRA improves both personalization and generalization in heterogeneous settings, making federated tuning of large language models more flexible and scalable.