How does feature learning change the function space evolution?

Feature learning is widely viewed as the mechanism that distinguishes neural networks from fixed-kernel methods, yet its effect on the underlying function space remains poorly understood. We aim to precisely characterize how the function space (e.g., RKHS) endowed by a two-layer neural network during gradient descent training. We prove that, in a high-dimensional proportional regime, the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance, yielding a deformed kernel that can be written as the original isotropic kernel evaluated on geometrically transformed inputs. This characterization enables a spectral analysis of the corresponding integral operator. We prove that the global eigenvalue decay rate is preserved, while the spike selectively alters the leading eigenspaces. For ReLU activations, we derive an explicit perturbative expansion showing that feature learning boosts the eigenvalue of the linear eigenfunction aligned with the target direction and mixes the top radial eigenfunction with a target-aligned quadratic harmonic. These results provide a precise function-space description of early feature learning: gradient descent does not merely rescale a static kernel, but induces a data-adaptive deformation that preferentially enriches directions aligned with the teacher signal.