LBI-FL: Low-Bit Integerized Federated Learning with Temporally Dynamic Bit-Width Allocation

Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50\% BitOPs per client on average for FL with less than 2\% accuracy loss compared to low-bit training with INT8 precision.

Training a shared AI model across many edge devices by sharing model without sharing data can protect privacy but require massive communication overheads of models and local model update on resource constrained edge devices.We introduce Low-Bit Integerized Federated Learning (LBI-FL), the first system that reduces all parts of the model — weights, activations, and gradients — from full-precision (32-bit) floating numbers to low-precision integer codes (fewer than eight bits). A lightweight reinforcement-learning agent watches the training process and assigns number of bits to each part iteration by iteration. It can dynamically allocate bit-widths based on the resulting accuracy loss. The agent is trained once on a toy dataset yet transfers effortlessly to new tasks and unbalanced data. LBI-FL can converge in training using low-precision integers as reliably as standard full-precision floating numbers. In practice, it cuts network traffic eight-fold and halves each device’s arithmetic work while keeping accuracy loss within two percentage. This makes privacy-preserving AI potentially feasible for phones, wearables, and other bandwidth-starved gadgets.