Timezone: »

 
Spotlight
Towards Domain-Agnostic Contrastive Learning
Vikas Verma · Thang Luong · Kenji Kawaguchi · Hieu Pham · Quoc Le

Tue Jul 20 05:25 PM -- 05:30 PM (PDT) @
in Deep Learning Algorithms 6 »

Despite recent successes, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a domain-agnostic approach to contrastive learning, named DACL, that is applicable to problems where domain-specific data augmentations are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. We theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning.

Keywords: Deep Learning

Author Information

Vikas Verma (Aalto University)
Thang Luong (Google Brain)
Kenji Kawaguchi (Harvard University)
Hieu Pham (Google)
Quoc Le (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors