Timezone: »
Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.
Author Information
Liam Collins (University of Texas at Austin)
Ph. D. student at UT Austin Electrical and Computer Engineering advised by Aryan Mokhtari and Sanjay Shakkottai. Princeton '19.
Aryan Mokhtari (UT Austin)
Sewoong Oh (University of Washington)
Sanjay Shakkottai (University of Texas at Austin)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: MAML and ANIL Provably Learn Representations »
Thu. Jul 21st through Fri the 22nd Room Hall E #632
More from the Same Authors
-
2021 : Minimax Optimization: The Case of Convex-Submodular »
Arman Adibi · Aryan Mokhtari · Hamed Hassani -
2023 : Improving multimodal datasets with image captioning »
Thao Nguyen · · Gabriel Ilharco · Sewoong Oh · Ludwig Schmidt -
2023 Poster: Collaborative Multi-Agent Heterogeneous Multi-Armed Bandits »
Ronshee Chawla · Daniel Vial · Sanjay Shakkottai · R Srikant -
2023 Poster: PAC Generalization via Invariant Representations »
Advait Parulekar · Karthikeyan Shanmugam · Sanjay Shakkottai -
2022 Poster: Asymptotically-Optimal Gaussian Bandits with Side Observations »
Alexia Atsidakou · Orestis Papadigenopoulos · Constantine Caramanis · Sujay Sanghavi · Sanjay Shakkottai -
2022 Spotlight: Asymptotically-Optimal Gaussian Bandits with Side Observations »
Alexia Atsidakou · Orestis Papadigenopoulos · Constantine Caramanis · Sujay Sanghavi · Sanjay Shakkottai -
2022 Poster: Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximation »
Daniel Vial · Advait Parulekar · Sanjay Shakkottai · R Srikant -
2022 Poster: De novo mass spectrometry peptide sequencing with a transformer model »
Melih Yilmaz · William Fondrie · Wout Bittremieux · Sewoong Oh · William Noble -
2022 Spotlight: Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximation »
Daniel Vial · Advait Parulekar · Sanjay Shakkottai · R Srikant -
2022 Spotlight: De novo mass spectrometry peptide sequencing with a transformer model »
Melih Yilmaz · William Fondrie · Wout Bittremieux · Sewoong Oh · William Noble -
2022 Poster: Linear Bandit Algorithms with Sublinear Time Complexity »
Shuo Yang · Tongzheng Ren · Sanjay Shakkottai · Eric Price · Inderjit Dhillon · Sujay Sanghavi -
2022 Poster: Sharpened Quasi-Newton Methods: Faster Superlinear Rate and Larger Local Convergence Neighborhood »
Qiujiang Jin · Alec Koppel · Ketan Rajawat · Aryan Mokhtari -
2022 Spotlight: Linear Bandit Algorithms with Sublinear Time Complexity »
Shuo Yang · Tongzheng Ren · Sanjay Shakkottai · Eric Price · Inderjit Dhillon · Sujay Sanghavi -
2022 Spotlight: Sharpened Quasi-Newton Methods: Faster Superlinear Rate and Larger Local Convergence Neighborhood »
Qiujiang Jin · Alec Koppel · Ketan Rajawat · Aryan Mokhtari -
2021 : Minimax Optimization: The Case of Convex-Submodular »
Hamed Hassani · Aryan Mokhtari · Arman Adibi -
2021 Poster: Exploiting Shared Representations for Personalized Federated Learning »
Liam Collins · Hamed Hassani · Aryan Mokhtari · Sanjay Shakkottai -
2021 Spotlight: Exploiting Shared Representations for Personalized Federated Learning »
Liam Collins · Hamed Hassani · Aryan Mokhtari · Sanjay Shakkottai -
2021 Poster: Defense against backdoor attacks via robust covariance estimation »
Jonathan Hayase · Weihao Kong · Raghav Somani · Sewoong Oh -
2021 Spotlight: Defense against backdoor attacks via robust covariance estimation »
Jonathan Hayase · Weihao Kong · Raghav Somani · Sewoong Oh -
2021 Poster: KO codes: inventing nonlinear encoding and decoding for reliable wireless communication via deep-learning »
Ashok Vardhan Makkuva · Xiyang Liu · Mohammad Vahid Jamali · Hessam Mahdavifar · Sewoong Oh · Pramod Viswanath -
2021 Spotlight: KO codes: inventing nonlinear encoding and decoding for reliable wireless communication via deep-learning »
Ashok Vardhan Makkuva · Xiyang Liu · Mohammad Vahid Jamali · Hessam Mahdavifar · Sewoong Oh · Pramod Viswanath -
2020 Poster: Quantized Decentralized Stochastic Learning over Directed Graphs »
Hossein Taheri · Aryan Mokhtari · Hamed Hassani · Ramtin Pedarsani -
2020 Poster: Optimal transport mapping via input convex neural networks »
Ashok Vardhan Makkuva · Amirhossein Taghvaei · Sewoong Oh · Jason Lee -
2020 Poster: InfoGAN-CR and ModelCentrality: Self-supervised Model Training and Selection for Disentangling GANs »
Zinan Lin · Kiran Thekumparampil · Giulia Fanti · Sewoong Oh -
2020 Poster: Meta-learning for Mixed Linear Regression »
Weihao Kong · Raghav Somani · Zhao Song · Sham Kakade · Sewoong Oh -
2019 Poster: Pareto Optimal Streaming Unsupervised Classification »
Soumya Basu · Steven Gutstein · Brent Lance · Sanjay Shakkottai -
2019 Oral: Pareto Optimal Streaming Unsupervised Classification »
Soumya Basu · Steven Gutstein · Brent Lance · Sanjay Shakkottai -
2018 Poster: Decentralized Submodular Maximization: Bridging Discrete and Continuous Settings »
Aryan Mokhtari · Hamed Hassani · Amin Karbasi -
2018 Oral: Decentralized Submodular Maximization: Bridging Discrete and Continuous Settings »
Aryan Mokhtari · Hamed Hassani · Amin Karbasi -
2018 Poster: Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communication »
Zebang Shen · Aryan Mokhtari · Tengfei Zhou · Peilin Zhao · Hui Qian -
2018 Oral: Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communication »
Zebang Shen · Aryan Mokhtari · Tengfei Zhou · Peilin Zhao · Hui Qian