SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Sai Praneeth Reddy Karimireddy · Satyen Kale · Mehryar Mohri · Sashank Jakkam Reddi · Sebastian Stich · Ananda Theertha Suresh

Keywords: [ Convex Optimization ] [ Non-convex Optimization ] [ Parallel and Distributed Learning ] [ Optimization - Convex ]

[ Abstract ] [ Join Zoom ]
Federated learning is a key scenario in modern large-scale machine learning where the data remains distributed over a large number of clients and the task is to learn a centralized model without transmitting the client data. The standard optimization algorithm used in this setting is Federated Averaging (FedAvg) due to its low communication cost. We obtain a tight characterization of the convergence of FedAvg and prove that heterogeneity (non-iid-ness) in the client's data results in a drift' in the local updates resulting in poor performance.