Timezone: »
Interacting systems are prevalent in nature, from dynamical systems in physics to complex societal dynamics. The interplay of components can give rise to complex behavior, which can often be explained using a simple model of the system's constituent parts. In this work, we introduce the neural relational inference (NRI) model: an unsupervised model that learns to infer interactions while simultaneously learning the dynamics purely from observational data. Our model takes the form of a variational auto-encoder, in which the latent code represents the underlying interaction graph and the reconstruction is based on graph neural networks. In experiments on simulated physical systems, we show that our NRI model can accurately recover ground-truth interactions in an unsupervised manner. We further demonstrate that we can find an interpretable structure and predict complex dynamics in real motion capture and sports tracking data.
Author Information
Thomas Kipf (University of Amsterdam)
Ethan Fetaya (University of Toronto)
Kuan-Chieh Wang (Univeristy of Toronto)
Max Welling (University of Amsterdam)
Richard Zemel (Vector Institute)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Neural Relational Inference for Interacting Systems »
Wed. Jul 11th 03:50 -- 04:00 PM Room A5
More from the Same Authors
-
2021 : Online Algorithmic Recourse by Collective Action »
Elliot Creager · Richard Zemel -
2022 : Towards Environment-Invariant Representation Learning for Robust Task Transfer »
Benjamin Eyre · Richard Zemel · Elliot Creager -
2022 : Path Integral Stochastic Optimal Control for Sampling Transition Paths »
Lars Holdijk · Yuanqi Du · Priyank Jaini · Ferry Hooft · Bernd Ensing · Max Welling -
2023 : Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift »
Benjamin Eyre · Elliot Creager · David Madras · Vardan Papyan · Richard Zemel -
2023 : Lie Point Symmetry and Physics Informed Networks »
Tara Akhound-Sadegh · Laurence Perreault-Levasseur · Johannes Brandstetter · Max Welling · Siamak Ravanbakhsh -
2023 Workshop: Structured Probabilistic Inference and Generative Modeling »
Dinghuai Zhang · Yuanqi Du · Chenlin Meng · Shawn Tan · Yingzhen Li · Max Welling · Yoshua Bengio -
2023 : Opening Remark »
Dinghuai Zhang · Yuanqi Du · Chenlin Meng · Shawn Tan · Yingzhen Li · Max Welling · Yoshua Bengio -
2023 Test Of Time: Learning Fair Representations »
Richard Zemel · Yu Wu · Kevin Swersky · Toniann Pitassi · Cynthia Dwork -
2023 Poster: Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames »
Ondrej Biza · Sjoerd van Steenkiste · Mehdi S. M. Sajjadi · Gamaleldin Elsayed · Aravindh Mahendran · Thomas Kipf -
2023 Poster: Scaling Vision Transformers to 22 Billion Parameters »
Mostafa Dehghani · Josip Djolonga · Basil Mustafa · Piotr Padlewski · Jonathan Heek · Justin Gilmer · Andreas Steiner · Mathilde Caron · Robert Geirhos · Ibrahim Alabdulmohsin · Rodolphe Jenatton · Lucas Beyer · Michael Tschannen · Anurag Arnab · Xiao Wang · Carlos Riquelme · Matthias Minderer · Joan Puigcerver · Utku Evci · Manoj Kumar · Sjoerd van Steenkiste · Gamaleldin Elsayed · Aravindh Mahendran · Fisher Yu · Avital Oliver · Fantine Huot · Jasmijn Bastings · Mark Collier · Alexey Gritsenko · Vighnesh N Birodkar · Cristina Vasconcelos · Yi Tay · Thomas Mensink · Alexander Kolesnikov · Filip Pavetic · Dustin Tran · Thomas Kipf · Mario Lucic · Xiaohua Zhai · Daniel Keysers · Jeremiah Harmsen · Neil Houlsby -
2023 Poster: Neural Wave Machines: Learning Spatiotemporally Structured Representations with Locally Coupled Oscillatory Recurrent Neural Networks »
T. Anderson Keller · Max Welling -
2023 Oral: Scaling Vision Transformers to 22 Billion Parameters »
Mostafa Dehghani · Josip Djolonga · Basil Mustafa · Piotr Padlewski · Jonathan Heek · Justin Gilmer · Andreas Steiner · Mathilde Caron · Robert Geirhos · Ibrahim Alabdulmohsin · Rodolphe Jenatton · Lucas Beyer · Michael Tschannen · Anurag Arnab · Xiao Wang · Carlos Riquelme · Matthias Minderer · Joan Puigcerver · Utku Evci · Manoj Kumar · Sjoerd van Steenkiste · Gamaleldin Elsayed · Aravindh Mahendran · Fisher Yu · Avital Oliver · Fantine Huot · Jasmijn Bastings · Mark Collier · Alexey Gritsenko · Vighnesh N Birodkar · Cristina Vasconcelos · Yi Tay · Thomas Mensink · Alexander Kolesnikov · Filip Pavetic · Dustin Tran · Thomas Kipf · Mario Lucic · Xiaohua Zhai · Daniel Keysers · Jeremiah Harmsen · Neil Houlsby -
2023 Poster: Latent Traversals in Generative Models as Potential Flows »
Yue Song · T. Anderson Keller · Nicu Sebe · Max Welling -
2023 Poster: Test-time Adaptation with Slot-Centric Models »
Mihir Prabhudesai · Anirudh Goyal · Sujoy Paul · Sjoerd van Steenkiste · Mehdi S. M. Sajjadi · Gaurav Aggarwal · Thomas Kipf · Deepak Pathak · Katerina Fragkiadaki -
2023 Poster: Geometric Clifford Algebra Networks »
David Ruhe · Jayesh K. Gupta · Steven De Keninck · Max Welling · Johannes Brandstetter -
2022 : Invited talks 3, Q/A, Amy, Rich and Liting »
Liting Sun · Amy Zhang · Richard Zemel -
2022 : Invited talks 3, Amy Zhang, Rich Zemel and Liting Sun »
Amy Zhang · Richard Zemel · Liting Sun -
2022 Poster: Lie Point Symmetry Data Augmentation for Neural PDE Solvers »
Johannes Brandstetter · Max Welling · Daniel Worrall -
2022 Spotlight: Lie Point Symmetry Data Augmentation for Neural PDE Solvers »
Johannes Brandstetter · Max Welling · Daniel Worrall -
2022 Poster: Equivariant Diffusion for Molecule Generation in 3D »
Emiel Hoogeboom · Victor Garcia Satorras · Clément Vignac · Max Welling -
2022 Oral: Equivariant Diffusion for Molecule Generation in 3D »
Emiel Hoogeboom · Victor Garcia Satorras · Clément Vignac · Max Welling -
2021 Test Of Time: Bayesian Learning via Stochastic Gradient Langevin Dynamics »
Yee Teh · Max Welling -
2021 Test Of Time: Test of Time Award »
Max Welling · Max Welling -
2021 Poster: SketchEmbedNet: Learning Novel Concepts by Imitating Drawings »
Alexander Wang · Mengye Ren · Richard Zemel -
2021 Poster: Learning a Universal Template for Few-shot Dataset Generalization »
Eleni Triantafillou · Hugo Larochelle · Richard Zemel · Vincent Dumoulin -
2021 Poster: Environment Inference for Invariant Learning »
Elliot Creager · Joern-Henrik Jacobsen · Richard Zemel -
2021 Spotlight: Learning a Universal Template for Few-shot Dataset Generalization »
Eleni Triantafillou · Hugo Larochelle · Richard Zemel · Vincent Dumoulin -
2021 Spotlight: Environment Inference for Invariant Learning »
Elliot Creager · Joern-Henrik Jacobsen · Richard Zemel -
2021 Spotlight: SketchEmbedNet: Learning Novel Concepts by Imitating Drawings »
Alexander Wang · Mengye Ren · Richard Zemel -
2021 Poster: The Hintons in your Neural Network: a Quantum Field Theory View of Deep Learning »
Roberto Bondesan · Max Welling -
2021 Spotlight: The Hintons in your Neural Network: a Quantum Field Theory View of Deep Learning »
Roberto Bondesan · Max Welling -
2021 Poster: A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups »
Marc Finzi · Max Welling · Andrew Wilson -
2021 Oral: A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups »
Marc Finzi · Max Welling · Andrew Wilson -
2021 Poster: Federated Learning of User Verification Models Without Sharing Embeddings »
Hossein Hosseini · Hyunsin Park · Sungrack Yun · Christos Louizos · Joseph B Soriaga · Max Welling -
2021 Poster: E(n) Equivariant Graph Neural Networks »
Victor Garcia Satorras · Emiel Hoogeboom · Max Welling -
2021 Poster: On Monotonic Linear Interpolation of Neural Network Parameters »
James Lucas · Juhan Bae · Michael Zhang · Stanislav Fort · Richard Zemel · Roger Grosse -
2021 Poster: Self Normalizing Flows »
T. Anderson Keller · Jorn Peters · Priyank Jaini · Emiel Hoogeboom · Patrick Forré · Max Welling -
2021 Spotlight: E(n) Equivariant Graph Neural Networks »
Victor Garcia Satorras · Emiel Hoogeboom · Max Welling -
2021 Spotlight: Federated Learning of User Verification Models Without Sharing Embeddings »
Hossein Hosseini · Hyunsin Park · Sungrack Yun · Christos Louizos · Joseph B Soriaga · Max Welling -
2021 Spotlight: Self Normalizing Flows »
T. Anderson Keller · Jorn Peters · Priyank Jaini · Emiel Hoogeboom · Patrick Forré · Max Welling -
2021 Spotlight: On Monotonic Linear Interpolation of Neural Network Parameters »
James Lucas · Juhan Bae · Michael Zhang · Stanislav Fort · Richard Zemel · Roger Grosse -
2020 : Invited Talk 4: Prof. Richard Zemel from University of Toronto »
Richard Zemel -
2020 : Invited talk 1: Unifying VAEs and Flows »
Max Welling -
2020 : Spotlight Talk 4: Few-shot Out-of-Distribution Detection »
Kuan-Chieh Wang -
2020 : Attentive Grouping and Graph Neural Networks for Object-Centric Learning »
Thomas Kipf -
2020 Workshop: Participatory Approaches to Machine Learning »
Angela Zhou · David Madras · Deborah Raji · Smitha Milli · Bogdan Kulynych · Richard Zemel -
2020 : Invited Talk: Thomas Kipf »
Thomas Kipf -
2020 Poster: Involutive MCMC: a Unifying Framework »
Kirill Neklyudov · Max Welling · Evgenii Egorov · Dmitry Vetrov -
2020 Poster: Causal Modeling for Fairness In Dynamical Systems »
Elliot Creager · David Madras · Toniann Pitassi · Richard Zemel -
2020 Poster: Optimizing Long-term Social Welfare in Recommender Systems: A Constrained Matching Approach »
Martin Mladenov · Elliot Creager · Omer Ben-Porat · Kevin Swersky · Richard Zemel · Craig Boutilier -
2020 Poster: Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling »
Will Grathwohl · Kuan-Chieh Wang · Joern-Henrik Jacobsen · David Duvenaud · Richard Zemel -
2019 Workshop: Learning and Reasoning with Graph-Structured Representations »
Ethan Fetaya · Zhiting Hu · Thomas Kipf · Yujia Li · Xiaodan Liang · Renjie Liao · Raquel Urtasun · Hao Wang · Max Welling · Eric Xing · Richard Zemel -
2019 : Poster discussion »
Roman Novak · Maxime Gabella · Frederic Dreyer · Siavash Golkar · Anh Tong · Irina Higgins · Mirco Milletari · Joe Antognini · Sebastian Goldt · Adín Ramírez Rivera · Roberto Bondesan · Ryo Karakida · Remi Tachet des Combes · Michael Mahoney · Nicholas Walker · Stanislav Fort · Samuel Smith · Rohan Ghosh · Aristide Baratin · Diego Granziol · Stephen Roberts · Dmitry Vetrov · Andrew Wilson · César Laurent · Valentin Thomas · Simon Lacoste-Julien · Dar Gilboa · Daniel Soudry · Anupam Gupta · Anirudh Goyal · Yoshua Bengio · Erich Elsen · Soham De · Stanislaw Jastrzebski · Charles H Martin · Samira Shabanian · Aaron Courville · Shorato Akaho · Lenka Zdeborova · Ethan Dyer · Maurice Weiler · Pim de Haan · Taco Cohen · Max Welling · Ping Luo · zhanglin peng · Nasim Rahaman · Loic Matthey · Danilo J. Rezende · Jaesik Choi · Kyle Cranmer · Lechao Xiao · Jaehoon Lee · Yasaman Bahri · Jeffrey Pennington · Greg Yang · Jiri Hron · Jascha Sohl-Dickstein · Guy Gur-Ari -
2019 : Panel Discussion (moderator: Tom Dietterich) »
Max Welling · Kilian Weinberger · Terrance Boult · Dawn Song · Thomas Dietterich -
2019 : Keynote by Max Welling: A Nonparametric Bayesian Approach to Deep Learning (without GPs) »
Max Welling -
2019 Workshop: Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (ODML-CDNNR) »
Sujith Ravi · Zornitsa Kozareva · Lixin Fan · Max Welling · Yurong Chen · Werner Bailer · Brian Kulis · Haoji Hu · Jonathan Dekhtiar · Yingyan Lin · Diana Marculescu -
2019 Workshop: Theoretical Physics for Deep Learning »
Jaehoon Lee · Jeffrey Pennington · Yasaman Bahri · Max Welling · Surya Ganguli · Joan Bruna -
2019 : Opening Remarks »
Jaehoon Lee · Jeffrey Pennington · Yasaman Bahri · Max Welling · Surya Ganguli · Joan Bruna -
2019 Poster: Stochastic Beams and Where To Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement »
Wouter Kool · Herke van Hoof · Max Welling -
2019 Poster: Lorentzian Distance Learning for Hyperbolic Representations »
Marc Law · Renjie Liao · Jake Snell · Richard Zemel -
2019 Poster: Flexibly Fair Representation Learning by Disentanglement »
Elliot Creager · David Madras · Joern-Henrik Jacobsen · Marissa Weis · Kevin Swersky · Toniann Pitassi · Richard Zemel -
2019 Oral: Lorentzian Distance Learning for Hyperbolic Representations »
Marc Law · Renjie Liao · Jake Snell · Richard Zemel -
2019 Oral: Flexibly Fair Representation Learning by Disentanglement »
Elliot Creager · David Madras · Joern-Henrik Jacobsen · Marissa Weis · Kevin Swersky · Toniann Pitassi · Richard Zemel -
2019 Oral: Stochastic Beams and Where To Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement »
Wouter Kool · Herke van Hoof · Max Welling -
2019 Poster: Understanding the Origins of Bias in Word Embeddings »
Marc-Etienne Brunet · Colleen Alkalay-Houlihan · Ashton Anderson · Richard Zemel -
2019 Poster: Emerging Convolutions for Generative Normalizing Flows »
Emiel Hoogeboom · Rianne Van den Berg · Max Welling -
2019 Oral: Understanding the Origins of Bias in Word Embeddings »
Marc-Etienne Brunet · Colleen Alkalay-Houlihan · Ashton Anderson · Richard Zemel -
2019 Oral: Emerging Convolutions for Generative Normalizing Flows »
Emiel Hoogeboom · Rianne Van den Berg · Max Welling -
2019 Poster: On the Universality of Invariant Networks »
Haggai Maron · Ethan Fetaya · Nimrod Segol · Yaron Lipman -
2019 Poster: Gauge Equivariant Convolutional Networks and the Icosahedral CNN »
Taco Cohen · Maurice Weiler · Berkay Kicanaoglu · Max Welling -
2019 Oral: Gauge Equivariant Convolutional Networks and the Icosahedral CNN »
Taco Cohen · Maurice Weiler · Berkay Kicanaoglu · Max Welling -
2019 Oral: On the Universality of Invariant Networks »
Haggai Maron · Ethan Fetaya · Nimrod Segol · Yaron Lipman -
2018 Poster: Attention-based Deep Multiple Instance Learning »
Maximilian Ilse · Jakub Tomczak · Max Welling -
2018 Poster: Learning Adversarially Fair and Transferable Representations »
David Madras · Elliot Creager · Toniann Pitassi · Richard Zemel -
2018 Oral: Attention-based Deep Multiple Instance Learning »
Maximilian Ilse · Jakub Tomczak · Max Welling -
2018 Oral: Learning Adversarially Fair and Transferable Representations »
David Madras · Elliot Creager · Toniann Pitassi · Richard Zemel -
2018 Poster: Reviving and Improving Recurrent Back-Propagation »
Renjie Liao · Yuwen Xiong · Ethan Fetaya · Lisa Zhang · Kijung Yoon · Zachary S Pitkow · Raquel Urtasun · Richard Zemel -
2018 Poster: Distilling the Posterior in Bayesian Neural Networks »
Kuan-Chieh Wang · Paul Vicol · James Lucas · Li Gu · Roger Grosse · Richard Zemel -
2018 Oral: Distilling the Posterior in Bayesian Neural Networks »
Kuan-Chieh Wang · Paul Vicol · James Lucas · Li Gu · Roger Grosse · Richard Zemel -
2018 Oral: Reviving and Improving Recurrent Back-Propagation »
Renjie Liao · Yuwen Xiong · Ethan Fetaya · Lisa Zhang · Kijung Yoon · Zachary S Pitkow · Raquel Urtasun · Richard Zemel -
2018 Invited Talk: Intelligence per Kilowatthour »
Max Welling -
2018 Poster: BOCK : Bayesian Optimization with Cylindrical Kernels »
ChangYong Oh · Efstratios Gavves · Max Welling -
2018 Oral: BOCK : Bayesian Optimization with Cylindrical Kernels »
ChangYong Oh · Efstratios Gavves · Max Welling -
2017 Poster: Deep Spectral Clustering Learning »
Marc Law · Raquel Urtasun · Richard Zemel -
2017 Poster: Multiplicative Normalizing Flows for Variational Bayesian Neural Networks »
Christos Louizos · Max Welling -
2017 Talk: Multiplicative Normalizing Flows for Variational Bayesian Neural Networks »
Christos Louizos · Max Welling -
2017 Talk: Deep Spectral Clustering Learning »
Marc Law · Raquel Urtasun · Richard Zemel