Florent Krzakala (EPFL), Asymptotics of Neural Networks in High Dimensions: From Sparse Representations to Deep Hierarchies

I will present recent advances in the asymptotic analysis of empirical risk minimization in overparameterized neural networks trained on synthetic Gaussian data, focusing on two key extensions of classical models: depth and nonlinearity. The aim is to understand what different architectures can learn and to characterize their inductive biases.

In the first part, I derive sharp asymptotic formulas for two-layer networks with quadratic activations and show that they exhibit a bias toward sparse representations, such as multi-index models. By mapping the learning dynamics to a convex matrix sensing problem with nuclear norm regularization, we obtain precise generalization thresholds and identify the fundamental limits of recovery.

In the second part, I analyze deep architectures trained on hierarchical Gaussian targets and show that depth enables effective dimensionality reduction. This leads to significantly improved sample complexity compared to kernel or shallow methods. A key requirement is that the target function be compositional and robust to noise at each level.

These results draw on tools from random matrix theory, convex optimization, and statistical physics, and help delineate the regimes in which deep learning offers provable advantages in high-dimensional settings.