Timezone: »

 
Spotlight
MAML and ANIL Provably Learn Representations
Liam Collins · Aryan Mokhtari · Sewoong Oh · Sanjay Shakkottai

Thu Jul 21 11:45 AM -- 11:50 AM (PDT) @ Room 310
in Probabilistic Methods/MISC »

Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.

Keywords: T: Optimization · MISC: Representation Learning · MISC: Transfer, Multitask and Meta-learning

Author Information

Liam Collins (University of Texas at Austin)

Ph. D. student at UT Austin Electrical and Computer Engineering advised by Aryan Mokhtari and Sanjay Shakkottai. Princeton BSE '19.

Aryan Mokhtari (UT Austin)
Sewoong Oh (University of Washington)
Sanjay Shakkottai (University of Texas at Austin)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors