Timezone: »

 
Which Features are Learned by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression
Yihao Xue · Siddharth Joshi · Eric Gan · Pin-Yu Chen · Baharan Mirzasoleiman

@
in The Second Workshop on Spurious Correlations, Invariance and Stability »
Event URL: https://openreview.net/forum?id=uzHHYTrl61 »

Contrastive learning (CL) has emerged as a powerful technique for representation learning, with or without label supervision.However, supervised CL is prone to collapsing representations of subclasses within a class by not capturing all their features, and unsupervised CL may suppress harder class-relevant features by focusing on learning easy class-irrelevant features; both significantly compromise representation quality. Yet, there is no theoretical understanding of \textit{class collapse} or \textit{feature suppression} at \textit{test} time. We provide the first unified theoretically rigorous framework to determine \textit{which} features are learnt by CL. Our analysis indicate that, perhaps surprisingly, bias of (stochastic) gradient descent towards finding simpler solutions is a key factor in collapsing subclass representations and suppressing harder class-relevant features. We also provide the first theoretical explanation for why employingsupervised and unsupervised CL together yields higher-quality representations, even when using commonly-used stochastic gradient methods.

Author Information

Yihao Xue (UCLA)
Siddharth Joshi (UCLA)
Eric Gan (UCLA)
Pin-Yu Chen (IBM Research)
Baharan Mirzasoleiman (UCLA)

More from the Same Authors