SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs

In large-scale LLM pretraining systems with $100\mathrm{k}+$ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, existing fault-tolerance mechanisms are largely unprepared for this restart-dominant regime. To address this challenge, we propose SPARe—Stacked Parallelism with Adaptive Reordering—a fault-tolerance framework that masks node failures during gradient synchronization by stacking redundant data shards across parallelism groups and adaptively reordering execution. SPARe achieves availability comparable to traditional replication while maintaining near-constant computation overhead of only $2\sim3\times$, even under high redundancy where traidional replication would require linearly inflating overhead. We derive closed-form expressions for endurable failure count and computation overhead, validate them via SimGrid-based discrete-event simulation, and jointly optimize redundancy and checkpointing to minimize training time. At extreme scale with up to $600\mathrm{k}$ GPUs, SPARe reduces time-to-train by $40\sim50$% compared to traditional replication.

SPARe is a failure masking scheme with redundant computation, yet retains near-constant computation overhead $2\sim3\times$ even for extreme redundancy degree such as r~20. Instead of fully replicating the entire parallelism groups, SPARe replicates shards of computation (data) and distributes across model-parallel groups with cyclic rotation and schedules gradient synchronization right after all types of partial gradients are collected. Upon failure accumulation, SPARe adaptively reorders the data shard stacks so that each training step can be completed with the minimal number of stacks computed. SPARe resides entirely on the data parallel layer, agnostic to the model architecture and inner parallelism topology, hence highly versatile to any distributed training system implementing synchronous data parallelism.