Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel

In this paper, we investigate deep feedforward neural networks with random weights. The input data matrix $\boldsymbol{X}$ is drawn from a Gaussian mixture model. We demonstrate that certain eigenvalues of the conjugate kernel and neural tangent kernel may lie outside the support of their limiting spectral measures in the high-dimensional regime. The existence and asymptotic positions of such isolated eigenvalues are rigorously analyzed. Furthermore, we provide a precise characterization of the entrywise limit of the projection matrix onto the eigenspace associated with these isolated eigenvalues. Our findings reveal that the eigenspace captures inherent group features present in $\boldsymbol{X}$. This study offers a quantitative analysis of how group features from the input data evolve through hidden layers in randomly weighted neural networks.

Imagine we have a large set of high-dimensional data—such as images or customer profiles—and we feed them into a neural network that has not been trained yet, with randomly initialized weights. Even at this untrained stage, the network transforms the data in meaningful ways as it propagates through each layer.In our paper, we focus on two important matrices in neural networks: the Conjugate Kernel (CK) and the Neural Tangent Kernel (NTK). These matrices play a central role in understanding how neural networks process and learn from data. Specifically, we investigate the presence of isolated eigenvalues in these matrices—eigenvalues that are clearly separated from the bulk of the spectrum. In methods like PCA, such isolated eigenvalues often correspond to meaningful patterns or hidden group structures in the data. This leads us to ask: can the eigenspaces corresponding to isolated eigenvalues of the CK and NTK matrices similarly capture useful information about the data?Our work demonstrates that, under mild conditions, isolated eigenvalues do indeed appear in the CK and NTK matrices. More importantly, we show that these eigenvalues and their associated eigenspaces contain valuable information about the underlying group structures in the data. We further support our theoretical findings with numerical simulations.