Statistical Consistency and Generalization of Contrastive Representation Learning

Contrastive representation learning (CRL) underpins many modern foundation models. Despite recent theoretical progress, existing analyses suffer from several key limitations: (i) the statistical consistency of CRL remains poorly understood; (ii) available generalization bounds deteriorate as the number of negative samples increases, contradicting the empirical benefits of large negative sets; and (iii) the retrieval performance of CRL has received limited theoretical attention. In this paper, we develop a unified statistical learning theory for CRL. For downstream tasks, we evaluate retrieval quality using an AUC-type population criterion and show that the contrastive loss is *statistically consistent* with optimal ranking. We further establish a *calibration-style inequality* that quantitatively relates excess contrastive risk to excess retrieval suboptimality. For upstream training, we study both supervised and self-supervised contrastive objectives and derive generalization bounds of order $O(1/m + 1/\sqrt{n})$ and $O(1/\sqrt{m} + 1/\sqrt{n})$, respectively, where $m$ denotes the number of negative samples and $n$ the number of anchor points. These bounds not only explain the empirical advantages of large negative sets but also reveal an explicit trade-off between $m$ and $n$. Extensive experiments on large-scale vision--language models corroborate our theoretical predictions.

Many modern AI systems learn useful representations by comparing related and unrelated examples—for example, matching an image with its correct caption while separating it from mismatched captions. This approach, called contrastive representation learning, is central to many foundation models, but its statistical foundations are still not fully understood. The first question we address is whether optimizing a contrastive training objective truly leads to optimal downstream retrieval behavior, such as ranking relevant items above irrelevant ones. We prove that it does: as contrastive risk approaches its optimum, retrieval performance also approaches the best possible retrieval performance. We further give a quantitative guarantee showing how improvements in contrastive training translate into improvements in downstream ranking. A second question is why using many negative examples helps in practice, even though some previous theory suggested it could hurt generalization. We prove new generalization bounds showing that more negative examples can improve learning, while also revealing when this benefit saturates relative to the number of anchor samples. Overall, our work gives a principled explanation for why contrastive learning can produce representations that transfer well to downstream tasks.