A Random Matrix Theory of Masked Self-Supervised Learning

In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik–Ben Arous–Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods.

In the era of sophisticated machine learning models, an increasingly used technique consists in hiding information in data, and using the dat itself to predict it. This is called masked self-supervised learning (masked, because some features are hidden, and self-supervised because the data itself is used to predict them). A theoretical analysis of this type of techniques is currently missing in the literature. In this paper, we provide a full theoretical analysis in a linear model, and study both its performance (as measure by a loss function) and the properties of the learned predictor.