Timezone: »

 
Spotlight
ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
Hui-Po Wang · Sebastian Stich · Yang He · Mario Fritz

Thu Jul 21 12:30 PM -- 12:35 PM (PDT) @ Room 327 - 329
in Optimization/Theory »
Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data. However, training is resource-intensive for edge devices, and limited network bandwidth is often the main bottleneck. Prior work often overcomes the constraints by condensing the models or messages into compact formats, e.g., by gradient compression or distillation. In contrast, we propose ProgFed, the first progressive training framework for efficient and effective federated learning. It inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. We theoretically prove that ProgFed converges at the same asymptotic rate as standard training on full models. Extensive results on a broad range of architectures, including CNNs (VGG, ResNet, ConvNets) and U-nets, and diverse tasks from simple classification to medical image segmentation show that our highly effective training approach saves up to $20\%$ computation and up to $63\%$ communication costs for converged models. As our approach is also complimentary to prior work on compression, we can achieve a wide range of trade-offs by combining these techniques, showing reduced communication of up to $50\times$ at only $0.1\%$ loss in utility. Code is available at https://github.com/a514514772/ProgFed.
Keywords: APP: Computer Vision · DL: Algorithms · OPT: Large Scale, Parallel and Distributed · MISC: Supervised Learning

Author Information

Hui-Po Wang (CISPA Helmholtz Center for Information Security)
Sebastian Stich (CISPA Helmholtz Center for Information Security gGmbH)
Yang He (Amazon)
Mario Fritz (CISPA Helmholtz Center for Infomation Security)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors