Timezone: »
Recent work in unsupervised representation learning has focused on learning deep directed latentvariable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.
Author Information
Alexander Alemi (Google)
Ben Poole (Stanford University)
Ian Fischer (Google)
Joshua V Dillon (Google)
Rif Saurous
Kevin Murphy (Google Brain)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Poster: Fixing a Broken ELBO »
Wed. Jul 11th 04:15 -- 07:00 PM Room Hall B #214
More from the Same Authors
-
2021 : A Closer Look at the Adversarial Robustness of Information Bottleneck Models »
Iryna Korshunova · David Stutz · Alexander Alemi · Olivia Wiles · Sven Gowal -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · Jie Ren · Joost van Amersfoort · Kehang Han · E. Kelly Buchanan · Kevin Murphy · Mark Collier · Mike Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2023 Poster: Sequential Monte Carlo Learning for Time Series Structure Discovery »
Feras Saad · Brian Patton · Matthew Hoffman · Rif Saurous · Vikash Mansinghka -
2022 Poster: Bayesian Imitation Learning for End-to-End Mobile Manipulation »
Yuqing Du · Daniel Ho · Alexander Alemi · Eric Jang · Mohi Khansari -
2022 Spotlight: Bayesian Imitation Learning for End-to-End Mobile Manipulation »
Yuqing Du · Daniel Ho · Alexander Alemi · Eric Jang · Mohi Khansari -
2020 Poster: The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks »
Jakub Swiatkowski · Kevin Roth · Bastiaan Veeling · Linh Tran · Joshua V Dillon · Jasper Snoek · Stephan Mandt · Tim Salimans · Rodolphe Jenatton · Sebastian Nowozin -
2020 Poster: Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems »
Zhe Dong · Bryan Seybold · Kevin Murphy · Hung Bui -
2020 Poster: Population-Based Black-Box Optimization for Biological Sequence Design »
Christof Angermueller · David Belanger · Andreea Gane · Zelda Mariet · David Dohan · Kevin Murphy · Lucy Colwell · D. Sculley -
2020 Poster: Weakly-Supervised Disentanglement Without Compromises »
Francesco Locatello · Ben Poole · Gunnar Ratsch · Bernhard Schölkopf · Olivier Bachem · Michael Tschannen -
2020 Poster: On Implicit Regularization in $\beta$-VAEs »
Abhishek Kumar · Ben Poole -
2019 Poster: Learning Latent Dynamics for Planning from Pixels »
Danijar Hafner · Timothy Lillicrap · Ian Fischer · Ruben Villegas · David Ha · Honglak Lee · James Davidson -
2019 Poster: On Variational Bounds of Mutual Information »
Ben Poole · Sherjil Ozair · Aäron van den Oord · Alexander Alemi · George Tucker -
2019 Oral: On Variational Bounds of Mutual Information »
Ben Poole · Sherjil Ozair · Aäron van den Oord · Alexander Alemi · George Tucker -
2019 Oral: Learning Latent Dynamics for Planning from Pixels »
Danijar Hafner · Timothy Lillicrap · Ian Fischer · Ruben Villegas · David Ha · Honglak Lee · James Davidson -
2019 Poster: NAS-Bench-101: Towards Reproducible Neural Architecture Search »
Chris Ying · Aaron Klein · Eric Christiansen · Esteban Real · Kevin Murphy · Frank Hutter -
2019 Oral: NAS-Bench-101: Towards Reproducible Neural Architecture Search »
Chris Ying · Aaron Klein · Eric Christiansen · Esteban Real · Kevin Murphy · Frank Hutter -
2018 Poster: Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron »
RJ Skerry-Ryan · Eric Battenberg · Ying Xiao · Yuxuan Wang · Daisy Stanton · Joel Shor · Ron Weiss · Robert Clark · Rif Saurous -
2018 Poster: Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis »
Yuxuan Wang · Daisy Stanton · Yu Zhang · RJ-Skerry Ryan · Eric Battenberg · Joel Shor · Ying Xiao · Ye Jia · Fei Ren · Rif Saurous -
2018 Oral: Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis »
Yuxuan Wang · Daisy Stanton · Yu Zhang · RJ-Skerry Ryan · Eric Battenberg · Joel Shor · Ying Xiao · Ye Jia · Fei Ren · Rif Saurous -
2018 Oral: Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron »
RJ Skerry-Ryan · Eric Battenberg · Ying Xiao · Yuxuan Wang · Daisy Stanton · Joel Shor · Ron Weiss · Robert Clark · Rif Saurous -
2017 Poster: Continual Learning Through Synaptic Intelligence »
Friedemann Zenke · Ben Poole · Surya Ganguli -
2017 Talk: Continual Learning Through Synaptic Intelligence »
Friedemann Zenke · Ben Poole · Surya Ganguli -
2017 Poster: On the Expressive Power of Deep Neural Networks »
Maithra Raghu · Ben Poole · Surya Ganguli · Jon Kleinberg · Jascha Sohl-Dickstein -
2017 Talk: On the Expressive Power of Deep Neural Networks »
Maithra Raghu · Ben Poole · Surya Ganguli · Jon Kleinberg · Jascha Sohl-Dickstein