Timezone: »
Transferability estimation has been an essential tool in selecting a pre-trained model and the layers in it for transfer learning, to transfer, so as to maximize the performance on a target task and prevent negative transfer. Existing estimation algorithms either require intensive training on target tasks or have difficulties in evaluating the transferability between layers. To this end, we propose a simple, efficient, and effective transferability measure named TransRate. Through a single pass over examples of a target task, TransRate measures the transferability as the mutual information between features of target examples extracted by a pre-trained model and their labels. We overcome the challenge of efficient mutual information estimation by resorting to coding rate that serves as an effective alternative to entropy. From the perspective of feature representation, the resulting TransRate evaluates both completeness (whether features contain sufficient information of a target task) and compactness (whether features of each class are compact enough for good generalization) of pre-trained features. Theoretically, we have analyzed the close connection of TransRate to the performance after transfer learning. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 35 pre-trained models and 16 downstream tasks.
Author Information
Long-Kai Huang (Tencent AI Lab)
Junzhou Huang (University of Texas at Arlington / Tencent AI Lab)
Yu Rong (Tencent AI Lab)
Qiang Yang (Hong Kong UST)
Ying WEI (City University of Hong Kong)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: Frustratingly Easy Transferability Estimation »
Tue. Jul 19th 02:35 -- 02:40 PM Room Room 318 - 320
More from the Same Authors
-
2022 Workshop: The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward »
Huaxiu Yao · Hugo Larochelle · Percy Liang · Colin Raffel · Jian Tang · Ying WEI · Saining Xie · Eric Xing · Chelsea Finn -
2022 : Hypergraph Convolutional Networks via Equivalence Between Hypergraphs and Undirected Graphs »
Jiying Zhang · fuyang li · Xi Xiao · Tingyang Xu · Yu Rong · Junzhou Huang · Yatao Bian -
2022 Poster: Local Augmentation for Graph Neural Networks »
Songtao Liu · Rex (Zhitao) Ying · Hanze Dong · Lanqing Li · Tingyang Xu · Yu Rong · Peilin Zhao · Junzhou Huang · Dinghao Wu -
2022 Spotlight: Local Augmentation for Graph Neural Networks »
Songtao Liu · Rex (Zhitao) Ying · Hanze Dong · Lanqing Li · Tingyang Xu · Yu Rong · Peilin Zhao · Junzhou Huang · Dinghao Wu -
2022 Poster: The Role of Deconfounding in Meta-learning »
Yinjie Jiang · Zhengyu Chen · Kun Kuang · Luotian Yuan · Xinhai Ye · Zhihua Wang · Fei Wu · Ying WEI -
2022 Spotlight: The Role of Deconfounding in Meta-learning »
Yinjie Jiang · Zhengyu Chen · Kun Kuang · Luotian Yuan · Xinhai Ye · Zhihua Wang · Fei Wu · Ying WEI -
2021 Poster: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Poster: Learning Diverse-Structured Networks for Adversarial Robustness »
Xuefeng Du · Jingfeng Zhang · Bo Han · Tongliang Liu · Yu Rong · Gang Niu · Junzhou Huang · Masashi Sugiyama -
2021 Spotlight: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Spotlight: Learning Diverse-Structured Networks for Adversarial Robustness »
Xuefeng Du · Jingfeng Zhang · Bo Han · Tongliang Liu · Yu Rong · Gang Niu · Junzhou Huang · Masashi Sugiyama -
2021 Poster: Meta-learning Hyperparameter Performance Prediction with Neural Processes »
Ying WEI · Peilin Zhao · Junzhou Huang -
2021 Spotlight: Meta-learning Hyperparameter Performance Prediction with Neural Processes »
Ying WEI · Peilin Zhao · Junzhou Huang -
2020 Poster: Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search »
Yong Guo · Yaofo Chen · Yin Zheng · Peilin Zhao · Jian Chen · Junzhou Huang · Mingkui Tan -
2020 Poster: Communication-Efficient Distributed PCA by Riemannian Optimization »
Long-Kai Huang · Jialin Pan -
2019 Poster: Hierarchically Structured Meta-learning »
Huaxiu Yao · Ying WEI · Junzhou Huang · Zhenhui (Jessie) Li -
2019 Poster: RaFM: Rank-Aware Factorization Machines »
Xiaoshuang Chen · Yin Zheng · Jiaxing Wang · Wenye Ma · Junzhou Huang -
2019 Oral: Hierarchically Structured Meta-learning »
Huaxiu Yao · Ying WEI · Junzhou Huang · Zhenhui (Jessie) Li -
2019 Oral: RaFM: Rank-Aware Factorization Machines »
Xiaoshuang Chen · Yin Zheng · Jiaxing Wang · Wenye Ma · Junzhou Huang -
2019 Poster: Collaborative Channel Pruning for Deep Networks »
Hanyu Peng · Jiaxiang Wu · Shifeng Chen · Junzhou Huang -
2019 Oral: Collaborative Channel Pruning for Deep Networks »
Hanyu Peng · Jiaxiang Wu · Shifeng Chen · Junzhou Huang -
2018 Poster: Adversarial Learning with Local Coordinate Coding »
Jiezhang Cao · Yong Guo · Qingyao Wu · Chunhua Shen · Junzhou Huang · Mingkui Tan -
2018 Poster: Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization »
Jiaxiang Wu · Weidong Huang · Junzhou Huang · Tong Zhang -
2018 Oral: Adversarial Learning with Local Coordinate Coding »
Jiezhang Cao · Yong Guo · Qingyao Wu · Chunhua Shen · Junzhou Huang · Mingkui Tan -
2018 Oral: Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization »
Jiaxiang Wu · Weidong Huang · Junzhou Huang · Tong Zhang -
2018 Poster: Transfer Learning via Learning to Transfer »
Ying WEI · Yu Zhang · Junzhou Huang · Qiang Yang -
2018 Oral: Transfer Learning via Learning to Transfer »
Ying WEI · Yu Zhang · Junzhou Huang · Qiang Yang