Timezone: »

 
Poster
Neural networks trained with SGD learn distributions of increasing complexity
Maria Refinetti · Alessandro Ingrosso · Sebastian Goldt

Thu Jul 27 04:30 PM -- 06:00 PM (PDT) @ Exhibit Hall 1 #442
in Poster Session 6 »

The uncanny ability of over-parameterised neural networks to generalise well has been explained using various "simplicity biases". These theories postulate that neural networks avoid overfitting by first fitting simple, linear classifiers before learning more complex, non-linear functions. Meanwhile, data structure is also recognised as a key ingredient for good generalisation, yet its role in simplicity biases is not yet understood. Here, we show that neural networks trained using stochastic gradient descent initially classify their inputs using lower-order input statistics, like mean and covariance, and exploit higher-order statistics only later during training. We first demonstrate this distributional simplicity bias (DSB) in a solvable model of a single neuron trained on synthetic data. We then demonstrate DSB empirically in a range of deep convolutional networks and visual transformers trained on CIFAR10, and show that it even holds in networks pre-trained on ImageNet. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of Gaussian universality in learning.

[ Slides]  [ PDF]  [ Poster] 

Author Information

Maria Refinetti (Laboratoire de Physique de l’Ecole Normale Supérieure Paris)
Alessandro Ingrosso (Abdus Salam international centre for theoretical physics)
Sebastian Goldt (International School of Advanced Studies (SISSA))

I'm an assistant professor working on theories of learning in neural networks.

More from the Same Authors