Timezone: »
Spotlight
Failure and success of the spectral bias prediction for Laplace Kernel Ridge Regression: the case of lowdimensional data
Umberto M. Tomasini · Antonio Sclocchi · Matthieu Wyart
Recently, several theories including the replica method made predictions for the generalization error of Kernel Ridge Regression. In some regimes, they predict that the method has a `spectral bias': decomposing the true function $f^*$ on the eigenbasis of the kernel, it fits well the coefficients associated with the O(P) largest eigenvalues, where $P$ is the size of the training set. This prediction works very well on benchmark data sets such as images, yet the assumptions these approaches make on the data are never satisfied in practice. To clarify when the spectral bias prediction holds, we first focus on a onedimensional model where rigorous results are obtained and then use scaling arguments to generalize and test our findings in higher dimensions. Our predictions include the classification case $f(x)=$sign$(x_1)$ with a data distribution that vanishes at the decision boundary $p(x)\sim x_1^{\chi}$. For $\chi>0$ and a Laplace kernel, we find that (i) there exists a crossover ridge $\lambda^*_{d,\chi}(P)\sim P^{\frac{1}{d+\chi}}$ such that for $\lambda\gg \lambda^*_{d,\chi}(P)$, the replica method applies, but not for $\lambda\ll\lambda^*_{d,\chi}(P)$, (ii) in the ridgeless case, spectral bias predicts the correct training curve exponent only in the limit $d\rightarrow\infty$.
Author Information
Umberto M. Tomasini (EPFL)
Theoretical Physicist by training, I am now at EPFL working on Machine Learning in the Matthieu Wyart Lab.
Antonio Sclocchi (EPFL  IPHYS  PCSL)
Matthieu Wyart
Related Events (a corresponding poster, oral, or spotlight)

2022 Poster: Failure and success of the spectral bias prediction for Laplace Kernel Ridge Regression: the case of lowdimensional data »
Wed. Jul 20th through Thu the 21st Room Hall E #1416
More from the Same Authors

2023 Poster: Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning »
Antonio Sclocchi · Mario Geiger · Matthieu Wyart 
2023 Poster: What can be learnt with wide convolutional neural networks? »
Francesco Cagnetta · Alessandro Favero · Matthieu Wyart 
2018 Poster: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco BaityJesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli 
2018 Oral: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco BaityJesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli