Timezone: »
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder, but we show how to drastically reduce this additional cost with couplings in the latent space. When parallel architectures can be exploited, our coders can achieve better rates than bits-back with little additional cost. We demonstrate improved lossless compression rates in a variety of settings, especially in out-of-distribution or sequential data compression.
Author Information
Yangjun Ruan (University of Toronto)
Karen Ullrich (FAIR)
Daniel Severo (University of Toronto)
James Townsend (UCL)
Ashish Khisti (Univ. of Toronto)
Arnaud Doucet (Oxford University)
Alireza Makhzani (University of Toronto)
Chris Maddison (University of Toronto)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Oral: Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding »
Wed. Jul 21st 12:00 -- 12:20 PM Room
More from the Same Authors
-
2022 : Learning to Discretize for Continuous-time Sequence Compression »
Ricky T. Q. Chen · Maximilian Nickel · Matthew Le · Matthew Muckley · Karen Ullrich -
2022 : Riemannian Diffusion Schr\"odinger Bridge »
James Thornton · Valentin De Bortoli · Michael Hutchinson · Emile Mathieu · Yee Whye Teh · Arnaud Doucet -
2022 : Contrastive Learning Can Find An Optimal Basis For Approximately Invariant Functions »
Daniel D. Johnson · Daniel D. Johnson · Ayoub El Hanchi · Ayoub El Hanchi · Chris Maddison · Chris Maddison -
2022 Poster: Augment with Care: Contrastive Learning for Combinatorial Problems »
Haonan Duan · Pashootan Vaezipoor · Max Paulus · Yangjun Ruan · Chris Maddison -
2022 Poster: Learning to Cut by Looking Ahead: Cutting Plane Selection via Imitation Learning »
Max Paulus · Giulia Zarpellon · Andreas Krause · Laurent Charlin · Chris Maddison -
2022 Spotlight: Augment with Care: Contrastive Learning for Combinatorial Problems »
Haonan Duan · Pashootan Vaezipoor · Max Paulus · Yangjun Ruan · Chris Maddison -
2022 Spotlight: Learning to Cut by Looking Ahead: Cutting Plane Selection via Imitation Learning »
Max Paulus · Giulia Zarpellon · Andreas Krause · Laurent Charlin · Chris Maddison -
2022 Poster: Bayesian Nonparametrics for Offline Skill Discovery »
Valentin Villecroze · Harry Braviner · Panteha Naderian · Chris Maddison · Gabriel Loaiza-Ganem -
2022 Spotlight: Bayesian Nonparametrics for Offline Skill Discovery »
Valentin Villecroze · Harry Braviner · Panteha Naderian · Chris Maddison · Gabriel Loaiza-Ganem -
2022 Poster: Stochastic Reweighted Gradient Descent »
Ayoub El Hanchi · David Stephens · Chris Maddison -
2022 Spotlight: Stochastic Reweighted Gradient Descent »
Ayoub El Hanchi · David Stephens · Chris Maddison -
2021 Poster: Monte Carlo Variational Auto-Encoders »
Achille Thin · Nikita Kotelevskii · Arnaud Doucet · Alain Durmus · Eric Moulines · Maxim Panov -
2021 Spotlight: Monte Carlo Variational Auto-Encoders »
Achille Thin · Nikita Kotelevskii · Arnaud Doucet · Alain Durmus · Eric Moulines · Maxim Panov -
2021 Poster: Differentiable Particle Filtering via Entropy-Regularized Optimal Transport »
Adrien Corenflos · James Thornton · George Deligiannidis · Arnaud Doucet -
2021 Oral: Differentiable Particle Filtering via Entropy-Regularized Optimal Transport »
Adrien Corenflos · James Thornton · George Deligiannidis · Arnaud Doucet -
2021 Poster: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions »
Will Grathwohl · Kevin Swersky · Milad Hashemi · David Duvenaud · Chris Maddison -
2021 Oral: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions »
Will Grathwohl · Kevin Swersky · Milad Hashemi · David Duvenaud · Chris Maddison -
2020 : Q&A: Chris Maddison »
Chris Maddison · Jessica Forde · Jesse Dodge -
2020 : Invited Talk: Chris Maddison »
Chris Maddison -
2020 Poster: Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows »
Rob Cornish · Anthony Caterini · George Deligiannidis · Arnaud Doucet -
2020 Poster: Evaluating Lossy Compression Rates of Deep Generative Models »
Sicong Huang · Alireza Makhzani · Yanshuai Cao · Roger Grosse -
2019 Poster: Replica Conditional Sequential Monte Carlo »
Alex Shestopaloff · Arnaud Doucet -
2019 Poster: Scalable Metropolis-Hastings for Exact Bayesian Inference with Large Datasets »
Rob Cornish · Paul Vanetti · Alexandre Bouchard-Côté · George Deligiannidis · Arnaud Doucet -
2019 Oral: Replica Conditional Sequential Monte Carlo »
Alex Shestopaloff · Arnaud Doucet -
2019 Oral: Scalable Metropolis-Hastings for Exact Bayesian Inference with Large Datasets »
Rob Cornish · Paul Vanetti · Alexandre Bouchard-Côté · George Deligiannidis · Arnaud Doucet -
2019 Poster: On the Impact of the Activation function on Deep Neural Networks Training »
Soufiane Hayou · Arnaud Doucet · Judith Rousseau -
2019 Oral: On the Impact of the Activation function on Deep Neural Networks Training »
Soufiane Hayou · Arnaud Doucet · Judith Rousseau