Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

We study the population loss landscape of two-layer ReLU networks of the form $\sum_{k=1}^K \mathrm{ReLU}(w_k^\top x)$ in a realisable teacher–student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.

Training a neural network means searching through many possible settings of its weights, and this search can get stuck in places that look good locally but are not the best solution. We study this problem in a controlled setting: a simple two-layer neural network learns a target function generated by another network of the same kind. This lets us replace the huge space of weights with a small set of summary quantities that exactly track the network's error and learning dynamics for Gaussian data. Using this reduced description, we identify hierarchical families of suboptimal solutions and explain why gradient-based training often ends at a small number of predictable error levels. We then show how adding extra hidden units changes which suboptimal solutions are stable and how often training reaches the correct solution. The picture is more nuanced than "larger networks always have an easy landscape": overparameterisation removes some traps while other, more structured traps can remain. These results provide a mathematical explanation of why extra model capacity can help optimisation, and a warning that common simplifications may miss important finite-size effects in neural-network training.