FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data. In practice, however, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training. We argue that a major source of this problem is *rotational misalignment*, arising from the rotational invariance of low-rank factorizations -- semantically equivalent updates can be represented in different latent subspaces across clients since $(B_i R_i)(R_i^\top A_i) = B_i A_i$. When such misaligned factors are averaged directly, they interfere destructively and degrade the global update. To address this issue, we propose **FedRot-LoRA**, a federated LoRA framework that aligns client updates via orthogonal transformations prior to aggregation. This alignment preserves the semantic update while reducing cross-client subspace mismatch, without increasing communication cost or restricting model expressivity. We provide a convergence analysis that examines the aggregation error induced by factor-wise averaging and shows how rotational alignment yields a tighter upper bound on this error. Extensive experiments on natural language understanding and generative tasks demonstrate that FedRot-LoRA consistently outperforms existing federated LoRA baselines across a range of heterogeneity levels and LoRA ranks. The code is available at https://github.com/haoran-zh/FedRot-LoRA.

Large language models often need to be adapted using data held by many organizations or devices, but privacy or legal constraints may prevent collecting that data in one place. Federated learning addresses this by letting each participant train locally and share only model updates, while LoRA makes these updates small enough to communicate efficiently. However, in federated LoRA, different participants may represent essentially the same update using different internal coordinate systems. When the server averages these mismatched pieces directly, useful information can interfere or cancel out, making training less stable and less accurate. We propose FedRot-LoRA, a method that first rotates each participant’s small model update into a shared orientation before averaging. This rotation keeps the meaning of each update unchanged while making the updates easier to combine. In experiments on language understanding, math reasoning, and code generation tasks, FedRot-LoRA improves training stability and performance compared with existing federated LoRA methods. This work can help organizations collaboratively adapt large AI models while keeping data local and preserving the communication efficiency needed for practical deployment.