FedQueue: Queue-Aware Federated Learning for Cross-Facility HPC Training

Federated learning~(FL) across multiple HPC facilities faces stochastic admission delays from batch schedulers that dominate wall-clock time. Synchronous FL suffers from severe stragglers, while asynchronous FL accumulates stale updates when queues spike. We propose FedQueue, a queue-aware FL protocol that incorporates scheduler delays directly into training and aggregation, which (i) predicts per-facility queue delays online to budget local work, (ii) applies cutoff-based admission that buffers late arrivals to bound staleness, and (iii) performs staleness-aware aggregation to stabilize heterogeneous local workloads. We prove the convergence for non-convex objectives at rate $\mathcal{O}(1/\sqrt{R})$ under bounded staleness, and show that the admission controls yield bounded staleness with high probability under queue-prediction error. Real-world cross-facility deployment of FedQueue shows 20.5\% improvement over baseline algorithms. Controlled queue simulations demonstrate robust improvement over the baselines; in particular, up to 60\% reduction in time to reach a target accuracy level under high queue variance and non-IID partitions.

Modern scientific AI models are growing so large that training them often requires combining computational resources from multiple supercomputing centers, but supercomputers do not run jobs on demand — every job waits in a queue, and these wait times can vary wildly from minutes to hours depending on how busy the system is. When several centers try to train a model together, these unpredictable delays badly disrupt the process: faster centers sit idle waiting for slower ones, or rush too far ahead. We built a training method called FedQueue that anticipates each center's queue wait and adjusts its workload accordingly, so no one is left idle and no one races too far ahead. When updates arrive late, we have rules for incorporating them carefully so the training stays stable, with mathematical guarantees that the model still converges. We tested FedQueue on four supercomputers in the US and found it trains models roughly 20% better than existing approaches, making it more practical to build scientific AI across institutions that cannot share raw data.