Timezone: »
Poster
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
Li Jing · Yichen Shen · Tena Dubcek · John E Peurifoy · Scott Skirlo · Yann LeCun · Max Tegmark · Marin Soljačić
Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. This approach appears particularly promising for Recurrent Neural Networks (RNNs). In this work, we present a new architecture for implementing an Efficient Unitary Neural Network (EUNNs); its main advantages can be summarized as follows. Firstly, the representation capacity of the unitary space in an EUNN is fully tunable, ranging from a subspace of SU(N) to the entire unitary space. Secondly, the computational complexity for training an EUNN is merely $\mathcal{O}(1)$ per parameter. Finally, we test the performance of EUNNs on the standard copying task, the pixel-permuted MNIST digit recognition benchmark as well as the Speech Prediction Test (TIMIT). We find that our architecture significantly outperforms both other state-of-the-art unitary RNNs and the LSTM architecture, in terms of the final performance and/or the wall-clock training speed. EUNNs are thus promising alternatives to RNNs and LSTMs for a wide variety of applications.
Author Information
Li Jing (Massachusetts Institute of Technology)
Yichen Shen (MIT)
Tena Dubcek (MIT)
John E Peurifoy (MIT)
Scott Skirlo (MIT)
Yann LeCun (New York University)
Max Tegmark (MIT)
Marin Soljačić (MIT)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Talk: Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs »
Tue. Aug 8th 01:06 -- 01:24 AM Room Parkside 1
More from the Same Authors
-
2022 : Deep Learning and Symbolic Regression for Discovering Parametric Equations »
Samuel Kim · Michael Zhang · Peter Y. Lu · Marin Soljačić -
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2022 : What Do We Maximize In Self-Supervised Learning? »
Ravid Shwartz-Ziv · Ravid Shwartz-Ziv · Randall Balestriero · Yann LeCun · Yann LeCun -
2023 Poster: Q-Flow: Generative Modeling for Differential Equations of Open Quantum Dynamics with Normalizing Flows »
Owen Dugan · Peter Y. Lu · Rumen Dangovski · Di Luo · Marin Soljačić -
2023 Poster: Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries »
Charlotte Loh · Seungwook Han · Shivchander Sudalairaj · Rumen Dangovski · Kai Xu · Florian Wenzel · Marin Soljačić · Akash Srivastava -
2023 Poster: PFGM++: Unlocking the Potential of Physics-Inspired Generative Models »
Yilun Xu · Ziming Liu · Yonglong Tian · Shangyuan Tong · Max Tegmark · Tommi Jaakkola -
2023 Poster: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank »
Quentin Garrido · Randall Balestriero · Laurent Najman · Yann LeCun -
2023 Poster: The SSL Interplay: Augmentations, Inductive Bias, and Generalization »
Vivien Cabannnes · Bobak T Kiani · Randall Balestriero · Yann LeCun · Alberto Bietti -
2023 Oral: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank »
Quentin Garrido · Randall Balestriero · Laurent Najman · Yann LeCun -
2023 Poster: Self-supervised learning of Split Invariant Equivariant representations »
Quentin Garrido · Laurent Najman · Yann LeCun -
2023 Poster: A Generalization of ViT/MLP-Mixer to Graphs »
Xiaoxin He · Bryan Hooi · Thomas Laurent · Adam Perold · Yann LeCun · Xavier Bresson -
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2018 Poster: Adversarially Regularized Autoencoders »
Jake Zhao · Yoon Kim · Kelly Zhang · Alexander Rush · Yann LeCun -
2018 Oral: Adversarially Regularized Autoencoders »
Jake Zhao · Yoon Kim · Kelly Zhang · Alexander Rush · Yann LeCun -
2018 Poster: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco Baity-Jesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli -
2018 Oral: Comparing Dynamics: Deep Neural Networks versus Glassy Systems »
Marco Baity-Jesi · Levent Sagun · Mario Geiger · Stefano Spigler · Gerard Arous · Chiara Cammarota · Yann LeCun · Matthieu Wyart · Giulio Biroli