Timezone: »

 
Spotlight
Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization
Jaehong Yoon · Geon Park · Wonyong Jeong · Sung Ju Hwang

Tue Jul 19 11:35 AM -- 11:40 AM (PDT) @ Ballroom 1 & 2
in DL: Algorithms »

In practical federated learning scenarios, the participating devices may have different bitwidths for computation and memory storage by design. However, despite the progress made in device-heterogeneous federated learning scenarios, the heterogeneity in the bitwidth specifications in the hardware has been mostly overlooked. We introduce a pragmatic FL scenario with bitwidth heterogeneity across the participating devices, dubbed as Bitwidth Heterogeneous Federated Learning (BHFL). BHFL brings in a new challenge, that the aggregation of model parameters with different bitwidths could result in severe performance degeneration, especially for high-bitwidth models. To tackle this problem, we propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights, and finally into full-precision weights. ProWD further selectively aggregates the model parameters to maximize the compatibility across bit-heterogeneous weights. We validate ProWD against relevant FL baselines on the benchmark datasets, using clients with varying bitwidths. Our ProWD largely outperforms the baseline FL algorithms as well as naive approaches (e.g. grouped averaging) under the proposed BHFL scenario.

Keywords: Deep Learning · DL: Algorithms · OPT: Large Scale, Parallel and Distributed · MISC: Transfer, Multitask and Meta-learning

Author Information

Jaehong Yoon (KAIST)
Geon Park (KAIST)
Wonyong Jeong (Korea Advanced Institute of Science and Technology)
Sung Ju Hwang (KAIST, AITRICS)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors