Timezone: »
Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.
Author Information
Aviv Navon (Bar-Ilan University)
Aviv Shamsian (Bar Ilan University)
Idan Achituve (Bar-Ilan)
Ethan Fetaya (Bar-Ilan University)
Gal Chechik (NVIDIA / Bar-Ilan University)
Haggai Maron (NVIDIA Research)
I am a Research Scientist at NVIDIA Research. My main fields of interest are machine learning, optimization, and shape analysis. More specifically, I am working on applying deep learning to irregular domains (e.g., graphs, point clouds, and surfaces) and graph/shape matching problems. I completed my Ph.D. in 2019 at the Department of Computer Science and Applied Mathematics at the Weizmann Institute of Science under the supervision of Prof. Yaron Lipman.
Related Events (a corresponding poster, oral, or spotlight)
-
2023 Poster: Equivariant Architectures for Learning in Deep Weight Spaces »
Wed. Jul 26th 09:00 -- 10:30 PM Room Exhibit Hall 1 #640
More from the Same Authors
-
2023 : Learning to Initiate and Reason in Event-Driven Cascading Processes »
Yuval Atzmon · Eli Meirom · Shie Mannor · Gal Chechik -
2023 : Expressive Sign Equivariant Networks for Spectral Geometric Learning »
Derek Lim · Joshua Robinson · Stefanie Jegelka · Haggai Maron -
2023 Oral: Equivariant Polynomials for Graph Neural Networks »
Omri Puny · Derek Lim · Bobak T Kiani · Haggai Maron · Yaron Lipman -
2023 Poster: Learning to Initiate and Reason in Event-Driven Cascading Processes »
Yuval Atzmon · Eli Meirom · Shie Mannor · Gal Chechik -
2023 Poster: Equivariant Polynomials for Graph Neural Networks »
Omri Puny · Derek Lim · Bobak T Kiani · Haggai Maron · Yaron Lipman -
2023 Poster: Graph Positional Encoding via Random Feature Propagation »
Moshe Eliasof · Fabrizio Frasca · Beatrice Bevilacqua · Eran Treister · Gal Chechik · Haggai Maron -
2023 Poster: Auxiliary Learning as an Asymmetric Bargaining Game »
Aviv Shamsian · Aviv Navon · Neta Glazer · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2022 Poster: Optimizing Tensor Network Contraction Using Reinforcement Learning »
Eli Meirom · Haggai Maron · Shie Mannor · Gal Chechik -
2022 Spotlight: Optimizing Tensor Network Contraction Using Reinforcement Learning »
Eli Meirom · Haggai Maron · Shie Mannor · Gal Chechik -
2022 Poster: Multi-Task Learning as a Bargaining Game »
Aviv Navon · Aviv Shamsian · Idan Achituve · Haggai Maron · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2022 Spotlight: Multi-Task Learning as a Bargaining Game »
Aviv Navon · Aviv Shamsian · Idan Achituve · Haggai Maron · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2021 Poster: GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning »
Idan Achituve · Aviv Navon · Yochai Yemini · Gal Chechik · Ethan Fetaya -
2021 Spotlight: GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning »
Idan Achituve · Aviv Navon · Yochai Yemini · Gal Chechik · Ethan Fetaya -
2021 Poster: Personalized Federated Learning using Hypernetworks »
Aviv Shamsian · Aviv Navon · Ethan Fetaya · Gal Chechik -
2021 Spotlight: Personalized Federated Learning using Hypernetworks »
Aviv Shamsian · Aviv Navon · Ethan Fetaya · Gal Chechik -
2021 Poster: Compositional Video Synthesis with Action Graphs »
Amir Bar · Roi Herzig · Xiaolong Wang · Anna Rohrbach · Gal Chechik · Trevor Darrell · Amir Globerson -
2021 Spotlight: Compositional Video Synthesis with Action Graphs »
Amir Bar · Roi Herzig · Xiaolong Wang · Anna Rohrbach · Gal Chechik · Trevor Darrell · Amir Globerson -
2021 Poster: Controlling Graph Dynamics with Reinforcement Learning and Graph Neural Networks »
Eli Meirom · Haggai Maron · Shie Mannor · Gal Chechik -
2021 Poster: From Local Structures to Size Generalization in Graph Neural Networks »
Gilad Yehudai · Ethan Fetaya · Eli Meirom · Gal Chechik · Haggai Maron -
2021 Spotlight: Controlling Graph Dynamics with Reinforcement Learning and Graph Neural Networks »
Eli Meirom · Haggai Maron · Shie Mannor · Gal Chechik -
2021 Spotlight: From Local Structures to Size Generalization in Graph Neural Networks »
Gilad Yehudai · Ethan Fetaya · Eli Meirom · Gal Chechik · Haggai Maron -
2020 Poster: On Learning Sets of Symmetric Elements »
Haggai Maron · Or Litany · Gal Chechik · Ethan Fetaya -
2020 Poster: Learning Algebraic Multigrid Using Graph Neural Networks »
Ilay Luz · Meirav Galun · Haggai Maron · Ronen Basri · Irad Yavneh