Feature Learning in High-Dimensions under Structured Covariance: Scaling Laws in Quadratic Networks

Recent theoretical work has shown that nonlinear solvable models exhibit scaling laws in the feature-learning regime. However, these results largely rely on the assumption of isotropic inputs, and understanding how these laws extend to anisotropic data remains a central open problem. In this work, we address this gap by analyzing the learning dynamics of two-layer neural networks with quadratic activations under anisotropic Gaussian inputs. We provide a sharp characterization of online stochastic gradient descent (SGD), explicitly quantifying how the covariance spectrum influences both the scaling exponent and sample complexity. Furthermore, we establish that normalization techniques overcome the intrinsic limitations of vanilla SGD, strictly improving sample efficiency in anisotropic settings. Experiments on two-layer networks with general activation functions support our theoretical predictions, suggesting that these insights extend well beyond the quadratic model.