Timezone: »

 
Poster
Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective
Liangliang Shi · Gu Zhang · Haoyu Zhen · Jintao Fan · Junchi Yan

Wed Jul 26 02:00 PM -- 03:30 PM (PDT) @ Exhibit Hall 1 #635
in Poster Session 3 »

Previous research on contrastive learning (CL) has primarily focused on pairwise views to learn representations by attracting positive samples and repelling negative ones. In this work, we aim to understand and generalize CL from a point set matching perspective, instead of the comparison between two points. Specifically, we formulate CL as a form of inverse optimal transport (IOT), which involves a bilevel optimization procedure for learning where the outter minimization aims to learn the representations and the inner is to learn the coupling (i.e. the probability of matching matrix) between the point sets. Specifically, by adjusting the relaxation degree of constraints in the inner minimization, we obtain three contrastive losses and show that the dominant contrastive loss in literature InfoNCE falls into one of these losses. This reveals a new and more general algorithmic framework for CL. Additionally, the soft matching scheme in IOT induces a uniformity penalty to enhance representation learning which is akin to the CL's uniformity. Results on vision benchmarks show the effectiveness of our derived loss family and the new uniformity term.

[ PDF]  [ Poster] 

Author Information

Liangliang Shi (Shanghai Jiao Tong University)
Gu Zhang (Shanghai Jiaotong University)
Haoyu Zhen (Shanghai Jiaotong University)
Jintao Fan (Shanghai Jiaotong University)
Junchi Yan (Shanghai Jiao Tong University)

More from the Same Authors