DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule

Advances in optimizers have been central to training large-scale neural networks, yet every modern optimizer still forms its momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate regardless of how often it is seen. Recent methods address the anisotropy of training only indirectly, wrapping extra processing around the momentum buffer while leaving the EMA update itself, the true source of this uniform forgetting, unchanged. We propose DeltaMomentum, which instead builds direction-awareness into the update rule. The gradient of a linear layer already splits into an input that acts as a key and an output-side error that acts as a value, and DeltaMomentum updates the momentum buffer by the classical delta rule, the standard update for key-value memories, so each direction is forgotten at a rate set by how often it appears. We prove that DeltaMomentum is a valid momentum for gradient descent, that it performs the input-side curvature correction of curvature-based methods with no matrix inversion, and that it clears stale directions faster than EMA both at a fixed optimum and under a drifting target. It is also a drop-in replacement for the momentum buffer in any optimizer, with a single added setting that transfers across model widths without re-tuning. On FineWeb-Edu language-model pretraining, DeltaMomentum reaches AdamW’s validation loss in up to 21% fewer steps at 67M and 21.55% at 370M and continues to a lower final loss, and training-time diagnostics confirm that the predicted mechanism, better gradient tracking and healthier input directions, drives the gain.