LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx10\times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.

Training modern, large-scale AI models requires linking multiple hardware accelerators together to share the workload. However, a major bottleneck exists in the amount of information, and thus time, needed to synchronize these accelerators. To reduce this overhead, researchers have developed algorithms that delay these synchronization points. However, this only solves one side of the problem (frequency); it doesn't reduce the amount of data sent at any single point. In fact, it actually increases it. To tackle this second issue, we introduce $\texttt{LoRDO}$. $\texttt{LoRDO}$ applies compression techniques along with this infrequent communication, allowing these systems to send *less information, less frequently*. When compared to traditional algorithms deployed in data centers, LoRDO achieves virtually the same performance while reducing communication overhead by roughly 10 times. This makes training large AI models much faster, cheaper, and more accessible, even on hardware with very limited memory.