Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs

Fine-tuning large language models (LLMs) on resource-constrained clients remains a challenging problem. Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with client model sizes and data scarcity. Still, the heterogeneity of resources remains a critical bottleneck: while higher-rank modules generally enhance performance, varying client capabilities constrain LoRA's feasible rank range. Existing approaches attempting to resolve this issue either lack analytical justification or impose additional computational overhead, leaving a wide gap for efficient and theoretically-grounded solutions. To address these challenges, we propose federated sketching LoRA (FSLoRA), which leverages a sketching mechanism to enable clients to selectively update submatrices of global LoRA modules maintained by the server. By adjusting the sketching ratios, which determine the ranks of the submatrices on the clients, FSLoRA flexibly adapts to client-specific communication and computational constraints. We provide a rigorous convergence analysis of FSLoRA that characterizes how the sketching ratios affect the convergence rate. Through extensive experiments, we demonstrate that FSLoRA outperforms baselines and significantly improves training efficiency while preserving stable convergence.

Training a large language model for a specific task usually requires a lot of computing power, which makes it hard to do across many personal devices like phones or laptops. This is especially difficult because some devices are much more powerful than others. Our method, called FSLoRA, helps devices work together more efficiently. Instead of asking every device to do the same amount of work, it lets each one update only a small part of the task adapter (i.e., LoRA) based on what it can handle. This lowers the computing and communication burden, so even weaker devices can still contribute. We also show mathematically that this approach remains reliable during training. In experiments, FSLoRA performed better than earlier methods while using resources more efficiently.