FedFit: Federated Dynamic Sparse Training via Fisher Information scoring

Cross-device Federated Learning (FL) is frequently bottlenecked by the prohibitive memory and communication costs of training deep neural networks on resource-constrained edge hardware. While federated dynamic sparse training aims to alleviate these costs by adjusting sparse structures during training, existing methods rely on magnitude-based heuristics that are fundamentally ill-suited for the non-convergent, heterogeneous environments inherent to FL. To address this challenge, we propose FedFit, a federated dynamic sparse training framework that replaces simple heuristics with optimization-centric criteria for structure adjustment. By leveraging a second-order approximation of the loss landscape via the Fisher Information Matrix, FedFit enables precise and efficient structure adjustment without the overhead of explicit Hessian computation. Empirical evaluations across computer vision and natural language processing benchmarks demonstrate that FedFit significantly narrows the sparse-to-dense accuracy gap, outperforming state-of-the-art methods while maintaining high communication efficiency. Our code is available at https://github.com/Serena-28/Fedfit.git.

Federated learning lets many devices, such as phones or sensors, learn together without sharing their private data. This is useful for privacy, but it is still hard to train large AI models on small devices because they have limited memory, computing power, and network bandwidth. A natural solution is to train only the most important parts of the model. The key question is how to decide which parts are important. Existing methods often make this choice using simple rules based on weight size, which can be unreliable when different devices have different data. FedFit helps devices make better choices about which parts of a model are worth training. In our experiments, this leads to better results than previous methods.