GreenMoE: Exploiting Dynamic Load Imbalance for Energy-Efficient Long-Context MoE Training

Large language model training increasingly combines long-context data with mixture-of-experts (MoE) architectures, but this setting also exposes a practical energy-efficiency problem: dynamic sequence packing and expert routing create substantial load imbalance across GPUs. Many energy-saving methods rely on static execution assumptions or reduce frequency on phases that may become critical, which can harm training throughput. We present GreenMoE, a dynamic-load-aware frequency scaling framework that treats non-critical-path waiting time as an energy-saving opportunity. GreenMoE estimates the load of Attention and MoE blocks at runtime, selects GPU core frequencies that satisfy a critical-path time constraint, and applies the selected frequencies through an asynchronous CUDA-event-based controller. In representative single-node configurations, GreenMoE reduces per-GPU step energy by 8.63% with 3.75% step-time overhead in a 64K setting and by up to 10.20% with larger overhead; in a constructed Attention-heavy setting, it reduces non-critical GPU energy by 16.15% with 0.23% overhead.