Green Gradients: A Loss-Landscape-Aware Hypergradient Meta-Optimizer for Eco-Friendly Transformer Training

Training large transformer models incurs substantial computational costs, with a significant portion arising from hyperparameter search. This research introduces Green Gradients, a dual-phase, calculus-based meta-optimizer designed to reduce the carbon footprint of transformer fine-tuning. Phase 1 replaces traditional expensive Bayesian learning rate search with a single exponential scan of the loss landscape, where $\partial\mathcal{L}/\partial\lambda$ is estimated to evaluate candidates and identify an optimal initial learning rate $\lambda^{*}$ based on the decrease in loss. Phase 2 adapts $\lambda$ throughout training via hypergradient descent by computing $\partial\mathcal{L}/\partial\lambda$ through the chain rule. Evaluated on DistilBERT applied to the CLIMATE-FEVER dataset across 25 paired trials, Green Gradients achieves a 48.87% reduction in both energy consumption and carbon emissions compared to an Optuna + ReduceLROnPlateau baseline ($p = 1.51\times10^{-11}$) with no statistically significant change in test accuracy ($p = 0.106$), demonstrating the viability of Green Gradients in reducing the carbon footprint of transformer training while retaining accuracy.