Timezone: »
The pre-train representation learning paradigm is a recent popular approach to address distribution shift and dataset limitation. This approach first pre-trains a representation function using large unlabeled datasets by self-supervised learning (e.g., contrastive learning), and then learns a classifier on the representation using small labeled datasets for downstream target tasks. The representation should have two key properties: label efficiency (i.e., learning an accurate classifier with a small amount of labeled data) and universality (i.e., useful for a wide range of downstream tasks). In this paper, we focus on contrastive learning and systematically study a trade-off between label efficiency and universality both empirically and theoretically. We empirically show that the trade-off exists in different models and datasets. Theoretically, we propose a data model with hidden representation and provide analysis in a simplified setting with linear models. The analysis shows that compared with pre-training on the target data directly, pre-training on diverse tasks can lead to a larger sample complexity for learning the classifier and thus worse prediction performance.
Author Information
Zhenmei Shi (UW-Madison)
Zhenmei Shi (UW-Madison)
Jiefeng Chen (University of Wisconsin-Madison)
Jiefeng Chen (University of Wisconsin-Madison)
Kunyang Li (University of Wisconsin - Madison)
Kunyang Li (University of Wisconsin - Madison)
Jayaram Raghuram (University of Wisconsin, Madison)
Jayaram Raghuram (University of Wisconsin, Madison)
Xi Wu (Google)
Completed my PhD in Computer Science from UW-Madison, advised by Jeffrey F. Naughton and Somesh Jha. Now a software engineer at Google. [Google PhD Fellow 2016 in privacy and security](https://ai.googleblog.com/2016/03/announcing-2016-google-phd-fellows-for.html).
Xi Wu (Google)
Completed my PhD in Computer Science from UW-Madison, advised by Jeffrey F. Naughton and Somesh Jha. Now a software engineer at Google. [Google PhD Fellow 2016 in privacy and security](https://ai.googleblog.com/2016/03/announcing-2016-google-phd-fellows-for.html).
Yingyiu Liang (University of Wisconsin-Madison)
Yingyiu Liang (University of Wisconsin-Madison)
Somesh Jha (University of Wisconsin, Madison)
Somesh Jha (University of Wisconsin, Madison)
More from the Same Authors
-
2021 : A Shuffling Framework For Local Differential Privacy »
Casey M Meehan · Amrita Roy Chowdhury · Kamalika Chaudhuri · Somesh Jha -
2023 Poster: When and How Does Known Class Help Discover Unknown Ones? Provable Understandings Through Spectral Analysis »
Yiyou Sun · Zhenmei Shi · Yingyiu Liang · Yixuan Li -
2023 Poster: Concept-based Explanations for Out-of-Distribution Detectors »
Jihye Choi · Jayaram Raghuram · Ryan Feng · Jiefeng Chen · Somesh Jha · Atul Prakash -
2023 Poster: Stratified Adversarial Robustness with Rejection »
Jiefeng Chen · Jayaram Raghuram · Jihye Choi · Xi Wu · Yingyiu Liang · Somesh Jha -
2022 : Adversarial Robustness and Cryptography »
Somesh Jha -
2021 Poster: A General Framework For Detecting Anomalous Inputs to DNN Classifiers »
Jayaram Raghuram · Varun Chandrasekaran · Somesh Jha · Suman Banerjee -
2021 Oral: A General Framework For Detecting Anomalous Inputs to DNN Classifiers »
Jayaram Raghuram · Varun Chandrasekaran · Somesh Jha · Suman Banerjee -
2021 Poster: Sample Complexity of Robust Linear Classification on Separated Data »
Robi Bhattacharjee · Somesh Jha · Kamalika Chaudhuri -
2021 Spotlight: Sample Complexity of Robust Linear Classification on Separated Data »
Robi Bhattacharjee · Somesh Jha · Kamalika Chaudhuri -
2020 Poster: Data-Dependent Differentially Private Parameter Learning for Directed Graphical Models »
Amrita Roy Chowdhury · Theodoros Rekatsinas · Somesh Jha -
2020 Poster: Concise Explanations of Neural Networks using Adversarial Training »
Prasad Chalasani · Jiefeng Chen · Amrita Roy Chowdhury · Xi Wu · Somesh Jha -
2020 Poster: CAUSE: Learning Granger Causality from Event Sequences using Attribution Methods »
Wei Zhang · Thomas Panum · Somesh Jha · Prasad Chalasani · David Page -
2019 Workshop: Workshop on the Security and Privacy of Machine Learning »
Nicolas Papernot · Florian Tramer · Bo Li · Dan Boneh · David Evans · Somesh Jha · Percy Liang · Patrick McDaniel · Jacob Steinhardt · Dawn Song -
2018 Poster: Analyzing the Robustness of Nearest Neighbors to Adversarial Examples »
Yizhen Wang · Somesh Jha · Kamalika Chaudhuri -
2018 Oral: Analyzing the Robustness of Nearest Neighbors to Adversarial Examples »
Yizhen Wang · Somesh Jha · Kamalika Chaudhuri -
2018 Poster: Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training »
Xi Wu · Wooyeong Jang · Jiefeng Chen · Lingjiao Chen · Somesh Jha -
2018 Oral: Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training »
Xi Wu · Wooyeong Jang · Jiefeng Chen · Lingjiao Chen · Somesh Jha