Decoupled Low-Rank Adaptation for Robust Federated Fine-Tuning

Federated Learning (FL) enables collaborative training across distributed clients while preserving data privacy. However, fine-tuning large-scale pre-trained models in FL is hindered by resource constraints and communication costs. Although introducing parameter-efficient fine-tuning strategies such as Low-Rank Adaptation (LoRA) effectively reduces trainable parameters, this low-rank constraint exacerbates noise sensitivity, leading to overfitting and aggregation bias. Existing robust federated fine-tuning methods rely on additional proxy data and treat low-rank adapters as generic weight vectors. In this paper, we investigate the structural properties of LoRA and reveal a robustness asymmetry. The down-projection matrix $A$ extracts stable general features, whereas the up-projection matrix $B$ is highly susceptible to fitting noise patterns. Based on this finding, we propose Federated Decoupled Robust LoRA (FedDR-LoRA), which employs a dual-branch mechanism to decouple robust feature learning from noise modeling and mitigates noise interference through noisy branch negative learning. During federated aggregation, we establish global consensus through aggregating $B$ while preserving local feature alignment in $A$. Extensive experiments demonstrate that FedDR-LoRA outperforms existing state-of-the-art methods across various noisy federated scenarios. Our code is available at: https://github.com/FangXiuwen/FedDR-LoRA.

Many AI models need to be adapted using data from different users, hospitals, companies, or devices, but directly sharing this data can raise privacy concerns. Federated learning addresses this by allowing different parties to train a model together without sending their raw data to a central server. However, adapting large pre-trained AI models in this setting is difficult because the devices may have limited computing power, and the local data may contain incorrect labels. This paper studies how to make this process more reliable when some training labels are wrong. We find that different parts of a lightweight model adaptation module behave differently under noisy labels: one part tends to learn stable general information, while another part is more likely to absorb incorrect label patterns. Based on this observation, we design FedDR-LoRA, a method that separates useful learning from noise modeling. One branch focuses on learning reliable information, while another branch helps identify and handle noisy samples. During collaboration across clients, the method shares useful knowledge while reducing the risk of spreading noise. Experiments on image and text classification tasks show that FedDR-LoRA is more robust than existing methods under various noisy data settings, including cases where different clients have different noise levels. This makes it a practical approach for adapting large AI models in privacy-preserving environments where data quality cannot be fully guaranteed.