Timezone: »

 
Spotlight
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Umesh Saunshi · Jordan Ash · Surbhi Goel · Dipendra Kumar Misra · Cyril Zhang · Sanjeev Arora · Sham Kakade · Akshay Krishnamurthy

Wed Jul 20 08:25 AM -- 08:30 AM (PDT) @ Room 309
in DL: Theory »

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of {\em augmentations} and the value of {\em contrastive loss} of representations. We demonstrate that such analyses, that ignore {\em inductive biases} of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even {\em provably} leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem -- different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.

Keywords: DL: Self-Supervised Learning · DL: Theory · T: Domain Adaptation and Transfer Learning · MISC: Representation Learning

Author Information

Nikunj Umesh Saunshi (Princeton University)
Jordan Ash (Microsoft Research)
Surbhi Goel (Microsoft Research)
Dipendra Kumar Misra (Microsoft Research)

Work -> Microsoft Research (2019 - ) PhD -> Cornell University (2019) in AI B-Tech -> IIT Kanpur (2013)

Cyril Zhang (Microsoft Research)
Sanjeev Arora (Princeton University)
Sham Kakade (Harvard University)
Akshay Krishnamurthy (Microsoft Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors