Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval

Spectral gradient methods, such as the Muon optimizer, modify gradient updates by preserving directional information while discarding scale, and have shown strong empirical performance in deep learning. We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model with anisotropic Gaussian inputs, equivalent to training a two-layer neural network with quadratic activation and fixed second-layer weights. Focusing on a spiked covariance setting where the dominant variance direction is orthogonal to the signal, we show that gradient descent (GD) suffers from a variance-induced misalignment: during the early escape phase, high-variance but uninformative directions are multiplicatively amplified, degrading alignment with the true signal under strong anisotropy. In contrast, spectral gradient descent (SpecGD) removes this anisotropic magnification effect, leading to stable alignment and accelerated noise contraction. Numerical experiments confirm the theory and show that these phenomena persist under more general anisotropic covariances.

Modern AI systems learn by continuously adjusting millions of parameters to extract structure from data. But when the data contain strong irrelevant variations, standard training methods can become distracted and focus on the wrong directions. In this work, we study this phenomenon through a mathematical model called phase retrieval, where the goal is to recover a hidden signal from indirect observations. We compare standard gradient descent with a spectral update inspired by the Muon optimizer, which removes distortions caused by uneven scaling in matrix-shaped updates. Our theory shows that this normalization prevents the algorithm from overreacting to noisy high-variance directions and helps it align with the true signal much earlier during training. As a result, the spectral method remains effective under much stronger anisotropy, suppresses irrelevant components faster, and reaches better solutions more reliably. Numerical experiments show that this advantage persists beyond the simplified theoretical setting, including in neural network training. These results help explain why matrix-aware optimizers such as Muon may better capture meaningful patterns in complex high-dimensional data.