A Solvable High-Dimensional Model Where Nonlinear Autoencoders Learn Structure Invisible to PCA While Test Loss Misaligns With Generalization

Many real-world datasets contain hidden structure that cannot be detected by simple linear correlations between input features. For example, latent factors may influence the data in a coordinated way, even though their effect is invisible to covariance-based methods such as PCA. In practice, nonlinear neural networks often succeed in extracting such hidden structure in unsupervised and self-supervised learning. However, constructing a minimal high-dimensional model where this advantage can be rigorously analyzed has remained an open theoretical challenge. We introduce a tractable high-dimensional spiked model with two latent factors: one visible to covariance, and one statistically dependent yet uncorrelated, appearing only in higher-order moments. PCA and linear autoencoders fail to recover the latter, while a minimal nonlinear autoencoder provably extracts both. We analyze both the population risk, and empirical risk minimization. Our model also provides a tractable example where self-supervised test loss is poorly aligned with representation quality: nonlinear autoencoders recover latent structure that linear methods miss, even though their reconstruction loss is higher.

Autoencoders are machine learning systems trained to compress their input into a learned representation and then use it to reconstruct the original data. Although this can be a useful way to learn without labels, it can also be misleading: a model may reconstruct its inputs accurately while missing patterns that are important for understanding the data. We study this issue in a simple mathematical setting where it is possible to track precisely which structures are learned. Our model contains two underlying signals. One is easy to identify with classical linear methods such as principal component analysis. The other is more subtle: it influences the data, but cannot be found by looking only at simple correlations. A useful analogy is that linear methods can only search along straight directions, whereas nonlinear methods can combine information in more flexible ways. In our model, this flexibility is essential: the subtle signal becomes visible only after the data are processed in the right nonlinear way. We show that even a very small nonlinear autoencoder can recover this structure, while a linear autoencoder cannot. At the same time, the linear autoencoder can still achieve lower reconstruction error. This gives a clear example where reproducing the data more accurately does not mean learning a better representation of its underlying structure. The broader message is that, in self-supervised learning, reconstruction error alone can give a misleading picture of what a model has actually learned.