Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participation

Partial participation is essential for communication-efficient federated learning at scale, yet existing Byzantine-robust methods typically assume full client participation. In the partial participation setting, a majority of the sampled clients may be Byzantine, once Byzantine clients dominate, existing methods break down immediately. We introduce delayed momentum aggregation, a principle where the central server aggregates cached momentum from non-sampled clients along with fresh momentum from sampled clients. This principle ensures Byzantine clients remain a minority from the server's perspective even when they dominate the sampled set. We instantiate this principle in our optimizer DeMoA. We analyze the convergence rate of DeMoA, showing that DeMoA is Byzantine-robust under partial participation. Experiments show that, with 20% Byzantine ratio and only 10% partial participation rate, DeMoA achieves the best accuracy even when existing methods fail empirically.

Imagine many hospitals, phones, or banks working together to train a single shared AI model without ever sharing their private data. This approach, called federated learning, lets each participant compute small updates from their own data and send only those updates to a central server, which combines them. To save on communication costs, the server usually only talks to a small random subset of participants in each round rather than all of them. A serious problem arises when some participants are dishonest or faulty. These participants are so-called "Byzantine" participants, and send corrupted updates designed to derail training. Existing defenses work only when honest participants form a clear majority in every round. But when the server contacts just a small subset at a time, by sheer bad luck the dishonest ones can temporarily outnumber the honest ones, and current methods break down. We propose a simple but powerful idea called delayed momentum aggregation. Instead of relying only on the few participants contacted this round, the server also remembers and reuses the most recent updates from everyone the server has heard from before. By blending these remembered updates with fresh ones, the dishonest participants are always outnumbered from the server's point of view, even if they happen to dominate the current sample. We build this idea into a new training algorithm DeMoA, prove mathematically that it converges reliably, and show in experiments that it keeps learning accurately in situations where previous methods fail.