Timezone: »

 
Leveraging Side Information for Communication-Efficient Federated Learning
Berivan Isik · Francesco Pase · Deniz Gunduz · Sanmi Koyejo · Tsachy Weissman · Michele Zorzi

@
in Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities »
Event URL: https://openreview.net/forum?id=1wakbAoVWo »
The high communication cost of sending model updates from the clients to the server is a significant bottleneck for scalable federated learning (FL). Among existing approaches, state-of-the-art bitrate-accuracy tradeoffs have been achieved using stochastic compression methods -- in which the client $n$ sends a sample from a client-only probability distribution $q_{\phi^{(n)}}$, and the server estimates the mean of the clients' distributions using these samples. However, such methods do not take full advantage of the FL setup where the server, throughout the training process, has \emph{side information} in the form of a pre-data distribution $p_{\theta}$ that is close to the client's distribution $q_{\phi^{(n)}}$ \emph{in Kullback–Leibler (KL) divergence}. We exploit this \emph{closeness} between the clients' distributions $q_{\phi^{(n)}}$'s and the side information $p_{\theta}$ at the server, and propose a framework that requires approximately $D_{KL}(q_{\phi^{(n)}}|| p_{\theta})$ bits of communication. We show that our method can be integrated into many existing stochastic compression frameworks such as FedPM, Federated SGLD, and QSGD to attain the same (and often higher) test accuracy with up to $50$ times reduction in the bitrate.

Author Information

Berivan Isik (Stanford University)
Francesco Pase (Universita' degli studi di Padova)
Deniz Gunduz (Imperial College London)
Sanmi Koyejo (Stanford University)
Tsachy Weissman (Stanford University)
Michele Zorzi (University of Padua)

More from the Same Authors