The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood. We develop a measure-theoretic framework formalizing the diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery. We show that in the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition and global minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non-orthogonal maps attain strictly lower asymptotic contrastive loss. We introduce a support-corrected Information Noise Contrastive Estimation (InfoNCE) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning.

Contrastive learning is a widely used approach for training machine learning models without manual labels. It works by bringing representations of related examples closer together while pushing unrelated examples apart. For example, two modified versions of the same image may be treated as related views of the same underlying object. This paper studies when this procedure can recover meaningful hidden structure in the data. Our main message is that the training loss alone is not enough. The way related examples are sampled is also crucial. We formalize this requirement through a diversity condition: the sampling process must vary the latent factors sufficiently for the model to recover their geometry. When this condition holds, contrastive learning can recover the latent space up to a distance-preserving transformation. When it fails, the contrastive objective may favor representations that achieve low loss while distorting the underlying structure. We also show that model architecture matters. Encoders with appropriate inductive biases can partially compensate when sampling diversity is limited. Synthetic experiments and CIFAR-10 experiments support this view, showing that representation quality depends jointly on the sampling mechanism and the architecture.