MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo

DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints. However, DiLoCo's performance has been shown to degrade as the number of workers (K) increases (Charles et al., 2025). In this work, we posit that a related but often overlooked factor in DiLoCo’s behavior is the choice of inner optimizer, which shapes the pseudogradient used by the outer optimizer. Given the recent success of Muon relative to AdamW for data parallel (DP) training, we examine how Muon's normalized optimizer steps can affect the pseudogradient's quality. We find that, relative to AdamW, Muon yields more *directionally correct* pseudogradients as the number of workers ($K$) increases. In our experiments pre-training language models, we conduct extensive hyperparameter tuning across 150M, 416M, 914M, 1.76B, and 3.1B models for DiLoCo, MuLoCo, AdamW DP, and Muon DP. Consistently across all scales, we find that with $K\geq1$ workers, MuLoCo (Muon inner optimizer DiLoCo) achieves superior performance to DiLoCo in absolute terms and for $K>2$ it outperforms DiLoCo relative to their data parallel baselines, while being compatible with quantization, streaming, and long synchronization intervals. At $K=1$, we find that MuLoCo can even outperform the data-parallel gold standard while having larger critical batch sizes and faster training times as a result.

Training large language models requires many computers to work together, but these computers often spend a lot of time communicating instead of learning. Existing methods such as DiLoCo reduce this communication by letting workers train independently for several steps before sharing updates, but their performance can worsen as more workers are added. In this work, we study whether changing the optimizer used inside DiLoCo can make this distributed training more reliable. We propose MuLoCo, which replaces DiLoCo’s standard AdamW optimizer with Muon, an optimizer whose updates are more consistently shaped across workers. We find that MuLoCo trains language models better than DiLoCo across model sizes, works better when many workers are used, and remains effective when communication is delayed, compressed, or streamed in pieces. We also find that MuLoCo can use much larger batch sizes, meaning it can make better use of many accelerators running in parallel. At 15-billion-parameter scale, MuLoCo remains competitive with strong data-parallel baselines while communicating less often. These results suggest that MuLoCo could make large language model training faster and more practical when computers are spread across slower or more limited networks.