Timezone: »
Probabilistic circuits (PCs) are a promising avenue for probabilistic modeling, as they permit a wide range of exact and efficient inference routines. Recent ``deep-learning-style'' implementations of PCs strive for a better scalability, but are still difficult to train on real-world data, due to their sparsely connected computational graphs. In this paper, we propose Einsum Networks (EiNets), a novel implementation design for PCs, improving prior art in several regards. At their core, EiNets combine a large number of arithmetic operations in a single monolithic einsum-operation, leading to speedups and memory savings of up to two orders of magnitude, in comparison to previous implementations. As an algorithmic contribution, we show that the implementation of Expectation-Maximization (EM) can be simplified for PCs, by leveraging automatic differentiation. Furthermore, we demonstrate that EiNets scale well to datasets which were previously out of reach, such as SVHN and CelebA, and that they can be used as faithful generative image models.
Author Information
Robert Peharz (Eindhoven University of Technology)
Steven Lang (Technical University of Darmstadt)
Antonio Vergari (University of California, Los Angeles)
Karl Stelzner (TU Darmstadt)
Alejandro Molina (TU Darmstadt)
Martin Trapp (Graz University of Technology)
Guy Van den Broeck (University of California, Los Angeles)
Kristian Kersting (TU Darmstadt)
Zoubin Ghahramani (University of Cambridge & Uber)
Zoubin Ghahramani is a Professor at the University of Cambridge, and Chief Scientist at Uber. He is also Deputy Director of the Leverhulme Centre for the Future of Intelligence, was a founding Director of the Alan Turing Institute and co-founder of Geometric Intelligence (now Uber AI Labs). His research focuses on probabilistic approaches to machine learning and AI. In 2015 he was elected a Fellow of the Royal Society.
More from the Same Authors
-
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · Jie Ren · Joost van Amersfoort · Kehang Han · E. Kelly Buchanan · Kevin Murphy · Mark Collier · Mike Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · JIE REN · Joost van Amersfoort · Kehang Han · Estefany Kelly Buchanan · Kevin Murphy · Mark Collier · Michael Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2023 : A Pseudo-Semantic Loss for Deep Generative Models with Logical Constraints »
Kareem Ahmed · Kai-Wei Chang · Guy Van den Broeck -
2023 : Collapsed Inference for Bayesian Deep Learning »
Zhe Zeng · Guy Van den Broeck -
2023 : SIMPLE: A Gradient Estimator for $k$-subset Sampling »
Kareem Ahmed · Zhe Zeng · Mathias Niepert · Guy Van den Broeck -
2023 : Probabilistic Task-Adaptive Graph Rewiring »
Chendi Qian · Andrei Manolache · Kareem Ahmed · Zhe Zeng · Guy Van den Broeck · Mathias Niepert · Christopher Morris -
2023 : A Unified Approach to Count-Based Weakly-Supervised Learning »
Vinay Shukla · Zhe Zeng · Kareem Ahmed · Guy Van den Broeck -
2023 : Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the Worlds Ugliness? »
Manuel Brack · Felix Friedrich · Patrick Schramowski · Kristian Kersting -
2023 : Panel on Reasoning Capabilities of LLMs »
Guy Van den Broeck · Ishita Dasgupta · Subbarao Kambhampati · Jiajun Wu · Xi Victoria Lin · Samy Bengio · Beliz Gunel -
2023 : AI can Learn from Data. But can it Learn to Reason? »
Guy Van den Broeck -
2023 Poster: ILLUME: Rationalizing Vision-Language Models through Human Interactions »
Manuel Brack · Patrick Schramowski · Björn Deiseroth · Kristian Kersting -
2023 Oral: Tractable Control for Autoregressive Language Generation »
Honghua Zhang · Meihua Dang · Nanyun Peng · Guy Van den Broeck -
2023 Poster: Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits »
Xuejie Liu · Anji Liu · Guy Van den Broeck · Yitao Liang -
2023 Poster: Tractable Control for Autoregressive Language Generation »
Honghua Zhang · Meihua Dang · Nanyun Peng · Guy Van den Broeck -
2023 Poster: Neural Diffusion Processes »
Vincent Dutordoir · Alan Saul · Zoubin Ghahramani · Fergus Simpson -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · JIE REN · Joost van Amersfoort · Kehang Han · Estefany Kelly Buchanan · Kevin Murphy · Mark Collier · Michael Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2022 : Session 3: New Computational Technologies for Reasoning »
Armando Solar-Lezama · Guy Van den Broeck · Jan-Willem van de Meent · Charles Sutton -
2022 Poster: Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks »
Lukas Struppek · Dominik Hintersdorf · Antonio De Almeida Correia · Antonia Adler · Kristian Kersting -
2022 Spotlight: Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks »
Lukas Struppek · Dominik Hintersdorf · Antonio De Almeida Correia · Antonia Adler · Kristian Kersting -
2021 Poster: Probabilistic Generating Circuits »
Honghua Zhang · Brendan Juba · Guy Van den Broeck -
2021 Oral: Probabilistic Generating Circuits »
Honghua Zhang · Brendan Juba · Guy Van den Broeck -
2021 Poster: Whittle Networks: A Deep Likelihood Model for Time Series »
Zhongjie Yu · Fabrizio Ventola · Kristian Kersting -
2021 Spotlight: Whittle Networks: A Deep Likelihood Model for Time Series »
Zhongjie Yu · Fabrizio Ventola · Kristian Kersting -
2020 : Invited Talk 6: Kristian Kersting (Q&A) »
Kristian Kersting -
2020 : Invited Talk 6: Kristian Kersting »
Kristian Kersting -
2020 : On the Relationship Between Probabilistic Circuits and Determinantal Point Processes »
Honghua Zhang · Steven Holtzen · Guy Van den Broeck -
2020 : Generative Adversarial Set Transformers »
Karl Stelzner -
2020 Poster: Scaling up Hybrid Probabilistic Inference with Logical and Arithmetic Constraints via Message Passing »
Zhe Zeng · Paolo Morettin · Fanqi Yan · Antonio Vergari · Guy Van den Broeck -
2019 : Sum-Product Networks and Deep Learning: A Love Marriage »
Robert Peharz -
2019 Workshop: The Third Workshop On Tractable Probabilistic Modeling (TPM) »
Pedro Domingos · Daniel Lowd · Tahrima Rahman · Antonio Vergari · Alejandro Molina · Antonio Vergari -
2019 Poster: Faster Attend-Infer-Repeat with Tractable Probabilistic Models »
Karl Stelzner · Robert Peharz · Kristian Kersting -
2019 Oral: Faster Attend-Infer-Repeat with Tractable Probabilistic Models »
Karl Stelzner · Robert Peharz · Kristian Kersting -
2019 Poster: Hierarchical Decompositional Mixtures of Variational Autoencoders »
Ping Liang Tan · Robert Peharz -
2019 Oral: Hierarchical Decompositional Mixtures of Variational Autoencoders »
Ping Liang Tan · Robert Peharz -
2018 Poster: Sound Abstraction and Decomposition of Probabilistic Programs »
Steven Holtzen · Guy Van den Broeck · Todd Millstein -
2018 Oral: Sound Abstraction and Decomposition of Probabilistic Programs »
Steven Holtzen · Guy Van den Broeck · Todd Millstein -
2018 Poster: Variational Bayesian dropout: pitfalls and fixes »
Jiri Hron · Alexander Matthews · Zoubin Ghahramani -
2018 Poster: The Mirage of Action-Dependent Baselines in Reinforcement Learning »
George Tucker · Surya Bhupatiraju · Shixiang Gu · Richard E Turner · Zoubin Ghahramani · Sergey Levine -
2018 Oral: Variational Bayesian dropout: pitfalls and fixes »
Jiri Hron · Alexander Matthews · Zoubin Ghahramani -
2018 Oral: The Mirage of Action-Dependent Baselines in Reinforcement Learning »
George Tucker · Surya Bhupatiraju · Shixiang Gu · Richard E Turner · Zoubin Ghahramani · Sergey Levine -
2018 Poster: Discovering Interpretable Representations for Both Deep Generative and Discriminative Models »
Tameem Adel · Zoubin Ghahramani · Adrian Weller -
2018 Poster: A Semantic Loss Function for Deep Learning with Symbolic Knowledge »
Jingyi Xu · Zilu Zhang · Tal Friedman · Yitao Liang · Guy Van den Broeck -
2018 Oral: Discovering Interpretable Representations for Both Deep Generative and Discriminative Models »
Tameem Adel · Zoubin Ghahramani · Adrian Weller -
2018 Oral: A Semantic Loss Function for Deep Learning with Symbolic Knowledge »
Jingyi Xu · Zilu Zhang · Tal Friedman · Yitao Liang · Guy Van den Broeck -
2017 Poster: Magnetic Hamiltonian Monte Carlo »
Nilesh Tripuraneni · Mark Rowland · Zoubin Ghahramani · Richard E Turner -
2017 Talk: Magnetic Hamiltonian Monte Carlo »
Nilesh Tripuraneni · Mark Rowland · Zoubin Ghahramani · Richard E Turner -
2017 Poster: Lost Relatives of the Gumbel Trick »
Matej Balog · Nilesh Tripuraneni · Zoubin Ghahramani · Adrian Weller -
2017 Poster: Bayesian inference on random simple graphs with power law degree distributions »
Juho Lee · Creighton Heaukulani · Zoubin Ghahramani · Lancelot F. James · Seungjin Choi -
2017 Talk: Lost Relatives of the Gumbel Trick »
Matej Balog · Nilesh Tripuraneni · Zoubin Ghahramani · Adrian Weller -
2017 Talk: Bayesian inference on random simple graphs with power law degree distributions »
Juho Lee · Creighton Heaukulani · Zoubin Ghahramani · Lancelot F. James · Seungjin Choi -
2017 Poster: Automatic Discovery of the Statistical Types of Variables in a Dataset »
Isabel Valera · Zoubin Ghahramani -
2017 Poster: A Birth-Death Process for Feature Allocation »
Konstantina Palla · David Knowles · Zoubin Ghahramani -
2017 Poster: Deep Bayesian Active Learning with Image Data »
Yarin Gal · Riashat Islam · Zoubin Ghahramani -
2017 Talk: A Birth-Death Process for Feature Allocation »
Konstantina Palla · David Knowles · Zoubin Ghahramani -
2017 Talk: Deep Bayesian Active Learning with Image Data »
Yarin Gal · Riashat Islam · Zoubin Ghahramani -
2017 Talk: Automatic Discovery of the Statistical Types of Variables in a Dataset »
Isabel Valera · Zoubin Ghahramani