Signal Frequency Imbalance and Ill-Conditioning

The source of the ill-conditioning addressed by Adam- and Muon-like optimizers remains poorly understood, making it unclear when and why they outperform SGD. We introduce a generalization of signal frequency imbalance that captures effective low-rank structure arising from correlations between weight-space directions and data subsets. Empirically, we show that this structure appears in the inner layers of language models as semantically meaningful gradient clusters, helping explain why Muon can outperform Adam. On a simplified problem, we show that signal frequency imbalance induces ill-conditioning only at large batch sizes, where minibatch averaging suppresses progress along rare directions, explaining why Adam and Muon outperform SGD only in this regime.