Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo

To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.

Currently, training large language models requires very expensive datacenters connected by ultra-fast networks. This puts cutting-edge AI out of reach for most researchers and concentrates power in a handful of organizations. Decentralized training methods instead aim to train large models across ordinary machines connected only by everyday internet links. The key issue is that communication between such machines is very slow. We build on two existing ideas: DiLoCo, which reduces how often machines need to communicate, and gossip, which reduces how many other machines each machine needs to communicate with. We combine these in a new framework. As part of that framework, we split communication into two parts based on whether it can be done at the same time as computation: an overlappable part and a non-overlappable part. This gives the flexibility to find the best balance between reducing communication that blocks computation and keeping enough communication for efficient training. In particular, we show configurations that make training practical and efficient at standard internet speeds.