Timezone: »

 
Poster
Universality of Winning Tickets: A Renormalization Group Perspective
William T. Redman · Tianlong Chen · Zhangyang “Atlas” Wang · Akshunna S. Dogra

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #429
in Poster Session 2 »

Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.

Keywords: DL: Algorithms · MISC: Transfer, Multitask and Meta-learning

Author Information

William T. Redman (UCSB)

I am a Dynamical Neuroscience (DYNS) PhD student at UC Santa Barbara, where I work in Prof. Michael Goard's systems neuroscience lab as a Chancellor's Fellow. I am also a researcher at AIMdyn, working under Prof. Igor Mezíc and Dr. Maria Fonoberova. Prior to being at UCSB, I got my bachelor's degrees in mathematics and physics from NYU. I am broadly interested in problems in neuroscience, Koopman operator theory, statistical physics, and machine learning.

Tianlong Chen (University of Texas at Austin)
Zhangyang “Atlas” Wang (University of Texas at Austin)
Akshunna S. Dogra (Imperial College London)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors