Identifying and Mitigating Errors in Gradient Aggregation of Distributed Data Parallel Training

Hardware-related silent data corruptions during gradient aggregation pose significant challenges to fault-tolerant distributed training, often leading to slow or failed convergence. To address this, we first mathematically formulate these errors as gradient inconsistency and theoretically analyze how they result in accumulated model divergence. Guided by this analysis, we introduce PAFT, a fault-tolerant distributed training system designed with dynamic and asynchronous parameter synchronization. PAFT comprises two core components: PAFT-Sync, which mitigates divergence via periodic synchronization, and PAFT-Dyn, which minimizes overhead through dynamic training overlap and frequency scheduling. Furthermore, the system’s synchronization mechanism is optimized to support standard optimizers, including SGD, SGD momentum, and Adam. We implement PAFT on PyTorch Distributed, and experimental results training ResNet, GPT-2, and LLaMA-2 on 4$\sim$32 GPUs demonstrate that it efficiently defends against aggregation errors while maintaining training performance.

Training modern AI models often requires many GPUs to work together, and these systems depend on frequent communication to combine gradients from different workers. In practice, silent hardware or network errors can corrupt this communication without causing the training job to crash. When that happens, different workers may update their models in inconsistent ways, gradually drifting apart and leading to slow training or even failure to converge. Our work studies this problem and shows, both mathematically and experimentally, how these hidden errors accumulate over time. To address it, we propose PAFT, a fault-tolerant training method that periodically re-synchronizes model parameters across workers. PAFT also adjusts how often this synchronization happens based on the observed level of model divergence, so it can respond more aggressively when errors are severe and save communication when errors are mild. We further design the synchronization to overlap with training in order to reduce overhead. Experiments on ResNet, GPT-2, and LLaMA-2 training show that PAFT can effectively recover from many gradient aggregation errors while preserving efficiency. This improves the reliability of large-scale AI training and can reduce wasted compute caused by hard-to-detect system faults.