Supervised-Contrastive Loss Learns Orthogonal Frames and Batching Matters

We are motivated by the question of what differences in the learning process occur when optimizing the supervised contrastive loss (SCL) and the cross-entropy (CE) loss. Our main finding is that the geometry of feature-embeddings learned by SCL forms an orthogonal frame (OF) regardless of the number of training examples per class. This is in contrast to the CE loss, for which previous work has shown that it learns embeddings geometries that are highly dependent on the class sizes. We arrive at our finding theoretically, by proving that the global minimizers of an unconstrained features model with SCL loss and entry-wise non-negativity constraints form an OF. We then validate the model's prediction by conducting experiments with standard deep-learning models on benchmark vision datasets. Finally, our analysis and experiments reveal that the batching scheme chosen during SCL training can play a critical role in speeding-up convergence to the OF geometry.