Timezone: »
This paper explores a new research problem of unsupervised transfer learning across multiple spatiotemporal prediction tasks. Unlike most existing transfer learning methods that focus on fixing the discrepancy between supervised tasks, we study how to transfer knowledge from a zoo of unsupervisedly learned models towards another predictive network. Our motivation is that models from different sources are expected to understand the complex spatiotemporal dynamics from different perspectives, thereby effectively supplementing the new task, even if the task has sufficient training samples. Technically, we propose a differentiable framework named transferable memory. It adaptively distills knowledge from a bank of memory states of multiple pretrained RNNs, and applies it to the target network via a novel recurrent structure called the Transferable Memory Unit (TMU). Compared with finetuning, our approach yields significant improvements on three benchmarks for spatiotemporal prediction, and benefits the target task even from less relevant pretext ones.
Author Information
Zhiyu Yao (Tsinghua University)
Yunbo Wang (Tsinghua University)
Mingsheng Long (Tsinghua University)
Jianmin Wang (Tsinghua University)
More from the Same Authors
-
2023 Poster: CLIPood: Generalizing CLIP to Out-of-Distributions »
Yang Shu · Xingzhuo Guo · Jialong Wu · Ximei Wang · Jianmin Wang · Mingsheng Long -
2023 Poster: Solving High-Dimensional PDEs with Latent Spectral Models »
Haixu Wu · Tengge Hu · huakun luo · Jianmin Wang · Mingsheng Long -
2023 Poster: Estimating Heterogeneous Treatment Effects: Mutual Information Bounds and Learning Algorithms »
Xingzhuo Guo · Yuchen Zhang · Jianmin Wang · Mingsheng Long -
2022 Poster: Flowformer: Linearizing Transformers with Conservation Flows »
Haixu Wu · Jialong Wu · Jiehui Xu · Jianmin Wang · Mingsheng Long -
2022 Poster: NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields »
Shanyan Guan · Huayu Deng · Yunbo Wang · Xiaokang Yang -
2022 Spotlight: Flowformer: Linearizing Transformers with Conservation Flows »
Haixu Wu · Jialong Wu · Jiehui Xu · Jianmin Wang · Mingsheng Long -
2022 Spotlight: NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields »
Shanyan Guan · Huayu Deng · Yunbo Wang · Xiaokang Yang -
2021 Poster: LogME: Practical Assessment of Pre-trained Models for Transfer Learning »
Kaichao You · Yong Liu · Jianmin Wang · Mingsheng Long -
2021 Spotlight: LogME: Practical Assessment of Pre-trained Models for Transfer Learning »
Kaichao You · Yong Liu · Jianmin Wang · Mingsheng Long -
2021 Poster: Representation Subspace Distance for Domain Adaptation Regression »
Xinyang Chen · Sinan Wang · Jianmin Wang · Mingsheng Long -
2021 Spotlight: Representation Subspace Distance for Domain Adaptation Regression »
Xinyang Chen · Sinan Wang · Jianmin Wang · Mingsheng Long -
2021 Poster: Self-Tuning for Data-Efficient Deep Learning »
Ximei Wang · Jinghan Gao · Mingsheng Long · Jianmin Wang -
2021 Poster: Zoo-Tuning: Adaptive Transfer from A Zoo of Models »
Yang Shu · Zhi Kou · Zhangjie Cao · Jianmin Wang · Mingsheng Long -
2021 Spotlight: Self-Tuning for Data-Efficient Deep Learning »
Ximei Wang · Jinghan Gao · Mingsheng Long · Jianmin Wang -
2021 Spotlight: Zoo-Tuning: Adaptive Transfer from A Zoo of Models »
Yang Shu · Zhi Kou · Zhangjie Cao · Jianmin Wang · Mingsheng Long -
2019 Poster: Bridging Theory and Algorithm for Domain Adaptation »
Yuchen Zhang · Tianle Liu · Mingsheng Long · Michael Jordan -
2019 Oral: Bridging Theory and Algorithm for Domain Adaptation »
Yuchen Zhang · Tianle Liu · Mingsheng Long · Michael Jordan -
2019 Poster: Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers »
Hong Liu · Mingsheng Long · Jianmin Wang · Michael Jordan -
2019 Poster: Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation »
Kaichao You · Ximei Wang · Mingsheng Long · Michael Jordan -
2019 Poster: Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation »
Xinyang Chen · Sinan Wang · Mingsheng Long · Jianmin Wang -
2019 Oral: Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation »
Kaichao You · Ximei Wang · Mingsheng Long · Michael Jordan -
2019 Oral: Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation »
Xinyang Chen · Sinan Wang · Mingsheng Long · Jianmin Wang -
2019 Oral: Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers »
Hong Liu · Mingsheng Long · Jianmin Wang · Michael Jordan -
2018 Poster: PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning »
Yunbo Wang · Zhifeng Gao · Mingsheng Long · Jianmin Wang · Philip Yu -
2018 Oral: PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning »
Yunbo Wang · Zhifeng Gao · Mingsheng Long · Jianmin Wang · Philip Yu -
2017 Poster: Deep Transfer Learning with Joint Adaptation Networks »
Mingsheng Long · Han Zhu · Jianmin Wang · Michael Jordan -
2017 Talk: Deep Transfer Learning with Joint Adaptation Networks »
Mingsheng Long · Han Zhu · Jianmin Wang · Michael Jordan