FedHera: Towards Drift-Resilient Federated Fine-tuning with Heterogeneous Resources

Driven by the imperative to leverage privacy-sensitive data scattered across decentralized devices, federated fine-tuning has emerged as a vital paradigm for adapting large language models without compromising data privacy. Yet, its practical efficacy is bottlenecked by severe client resource heterogeneity. Existing truncation-based methods typically couple the transmitted rank with the trainable rank, which (i) under-utilizes bandwidth on communication-rich but compute-limited clients and (ii) exacerbates truncation-induced gradient drift. To address this, we propose FedHera, a resource-decoupled framework that explicitly differentiates information reception from gradient optimization. FedHera employs a spectrum-preserving allocation strategy to maximize the transfer of global knowledge (via high-rank singular values) within bandwidth limits, irrespective of training constraints. Furthermore, we introduce a prefix-gating mechanism that utilizes the downloaded high-capacity basis as a frozen reference to guide local updates, thereby minimizing the optimization gap caused by aggressive truncation. Extensive experiments under different heterogeneous settings show that FedHera improves stability and accuracy over state-of-the-art baselines.

Large language models often need to be adapted using data that is spread across many users, organizations, or devices. For privacy reasons, this data may not be collected in one central place. Federated learning offers a way for different devices to collaborate without sharing their raw data, but this becomes difficult when the devices have very different capabilities. Some devices may have a good internet connection and can receive a larger model update, but they may not have enough memory or computing power to fully train it. This paper introduces FedHera, a method for making federated adaptation of large language models more reliable under such uneven device resources. The key idea is to separate what a device can receive from what it can train. A device can receive a richer version of the shared model information, while only training the part that fits within its local resource limits. The remaining received information is kept as a stable reference during training, helping each device stay better aligned with the shared learning process. Experiments on several language understanding and generation tasks show that this design improves both model quality and training stability compared with existing approaches. Overall, FedHera helps make privacy-preserving adaptation of large language models more practical when participating devices have different communication, memory, and computing capacities.