Timezone: »
Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as natural images, artwork, and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which directly encodes from and decodes to these parse trees, ensuring the generated outputs are always syntactically valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecule generation.
Author Information
Matt J. Kusner (Alan Turing Institute)
Brooks Paige (Alan Turing Institute)
Jose Miguel Hernandez-Lobato (University of Cambridge)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Talk: Grammar Variational Autoencoder »
Wed. Aug 9th 06:24 -- 06:42 AM Room Parkside 1
More from the Same Authors
-
2023 : Leveraging Task Structures for Improved Identifiability in Neural Network Representations »
Wenlin Chen · Julien Horwood · Juyeon Heo · Jose Miguel Hernandez-Lobato -
2023 : Minimal Random Code Learning with Mean-KL Parameterization »
Jihao Andreas Lin · Gergely Flamich · Jose Miguel Hernandez-Lobato -
2022 Poster: Adapting the Linearised Laplace Model Evidence for Modern Deep Learning »
Javier Antorán · David Janz · James Allingham · Erik Daxberger · Riccardo Barbano · Eric Nalisnick · Jose Miguel Hernandez-Lobato -
2022 Spotlight: Adapting the Linearised Laplace Model Evidence for Modern Deep Learning »
Javier Antorán · David Janz · James Allingham · Erik Daxberger · Riccardo Barbano · Eric Nalisnick · Jose Miguel Hernandez-Lobato -
2022 Poster: Action-Sufficient State Representation Learning for Control with Structural Constraints »
Biwei Huang · Chaochao Lu · Liu Leqi · Jose Miguel Hernandez-Lobato · Clark Glymour · Bernhard Schölkopf · Kun Zhang -
2022 Spotlight: Action-Sufficient State Representation Learning for Control with Structural Constraints »
Biwei Huang · Chaochao Lu · Liu Leqi · Jose Miguel Hernandez-Lobato · Clark Glymour · Bernhard Schölkopf · Kun Zhang -
2022 Poster: Fast Relative Entropy Coding with A* coding »
Gergely Flamich · Stratis Markou · Jose Miguel Hernandez-Lobato -
2022 Spotlight: Fast Relative Entropy Coding with A* coding »
Gergely Flamich · Stratis Markou · Jose Miguel Hernandez-Lobato -
2021 Poster: Operationalizing Complex Causes: A Pragmatic View of Mediation »
Limor Gultchin · David Watson · Matt J. Kusner · Ricardo Silva -
2021 Poster: Active Slices for Sliced Stein Discrepancy »
Wenbo Gong · Kaibo Zhang · Yingzhen Li · Jose Miguel Hernandez-Lobato -
2021 Spotlight: Operationalizing Complex Causes: A Pragmatic View of Mediation »
Limor Gultchin · David Watson · Matt J. Kusner · Ricardo Silva -
2021 Spotlight: Active Slices for Sliced Stein Discrepancy »
Wenbo Gong · Kaibo Zhang · Yingzhen Li · Jose Miguel Hernandez-Lobato -
2021 Poster: Learning Binary Decision Trees by Argmin Differentiation »
Valentina Zantedeschi · Matt J. Kusner · Vlad Niculae -
2021 Poster: A Gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter Optimization »
Andrew Campbell · Wenlong Chen · Vincent Stimper · Jose Miguel Hernandez-Lobato · Yichuan Zhang -
2021 Spotlight: Learning Binary Decision Trees by Argmin Differentiation »
Valentina Zantedeschi · Matt J. Kusner · Vlad Niculae -
2021 Spotlight: A Gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter Optimization »
Andrew Campbell · Wenlong Chen · Vincent Stimper · Jose Miguel Hernandez-Lobato · Yichuan Zhang -
2021 Poster: Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction »
Afsaneh Mastouri · Yuchen Zhu · Limor Gultchin · Anna Korba · Ricardo Silva · Matt J. Kusner · Arthur Gretton · Krikamol Muandet -
2021 Spotlight: Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction »
Afsaneh Mastouri · Yuchen Zhu · Limor Gultchin · Anna Korba · Ricardo Silva · Matt J. Kusner · Arthur Gretton · Krikamol Muandet -
2021 Poster: Bayesian Deep Learning via Subnetwork Inference »
Erik Daxberger · Eric Nalisnick · James Allingham · Javier Antorán · Jose Miguel Hernandez-Lobato -
2021 Spotlight: Bayesian Deep Learning via Subnetwork Inference »
Erik Daxberger · Eric Nalisnick · James Allingham · Javier Antorán · Jose Miguel Hernandez-Lobato -
2020 : "Latent Space Optimization with Deep Generative Models" »
Jose Miguel Hernandez-Lobato -
2020 : Invited Talk: Efficient Missing-value Acquisition with Variational Autoencoders »
Jose Miguel Hernandez-Lobato -
2020 Poster: Reinforcement Learning for Molecular Design Guided by Quantum Mechanics »
Gregor Simm · Robert Pinsler · Jose Miguel Hernandez-Lobato -
2020 Poster: A Generative Model for Molecular Distance Geometry »
Gregor Simm · Jose Miguel Hernandez-Lobato -
2019 Poster: Dropout as a Structured Shrinkage Prior »
Eric Nalisnick · Jose Miguel Hernandez-Lobato · Padhraic Smyth -
2019 Poster: Making Decisions that Reduce Discriminatory Impacts »
Matt J. Kusner · Chris Russell · Joshua Loftus · Ricardo Silva -
2019 Oral: Dropout as a Structured Shrinkage Prior »
Eric Nalisnick · Jose Miguel Hernandez-Lobato · Padhraic Smyth -
2019 Oral: Making Decisions that Reduce Discriminatory Impacts »
Matt J. Kusner · Chris Russell · Joshua Loftus · Ricardo Silva -
2019 Poster: EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE »
Chao Ma · Sebastian Tschiatschek · Konstantina Palla · Jose Miguel Hernandez-Lobato · Sebastian Nowozin · Cheng Zhang -
2019 Poster: Variational Implicit Processes »
Chao Ma · Yingzhen Li · Jose Miguel Hernandez-Lobato -
2019 Oral: Variational Implicit Processes »
Chao Ma · Yingzhen Li · Jose Miguel Hernandez-Lobato -
2019 Oral: EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE »
Chao Ma · Sebastian Tschiatschek · Konstantina Palla · Jose Miguel Hernandez-Lobato · Sebastian Nowozin · Cheng Zhang -
2018 Poster: Blind Justice: Fairness with Encrypted Sensitive Attributes »
Niki Kilbertus · Adria Gascon · Matt Kusner · Michael Veale · Krishna Gummadi · Adrian Weller -
2018 Oral: Blind Justice: Fairness with Encrypted Sensitive Attributes »
Niki Kilbertus · Adria Gascon · Matt Kusner · Michael Veale · Krishna Gummadi · Adrian Weller -
2018 Poster: TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service »
Amartya Sanyal · Matt Kusner · Adria Gascon · Varun Kanade -
2018 Poster: Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning »
Stefan Depeweg · Jose Miguel Hernandez-Lobato · Finale Doshi-Velez · Steffen Udluft -
2018 Oral: Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning »
Stefan Depeweg · Jose Miguel Hernandez-Lobato · Finale Doshi-Velez · Steffen Udluft -
2018 Oral: TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service »
Amartya Sanyal · Matt Kusner · Adria Gascon · Varun Kanade -
2017 Poster: Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space »
Jose Miguel Hernandez-Lobato · James Requeima · Edward Pyzer-Knapp · Alan Aspuru-Guzik -
2017 Talk: Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space »
Jose Miguel Hernandez-Lobato · James Requeima · Edward Pyzer-Knapp · Alan Aspuru-Guzik -
2017 Poster: Sequence Tutor: Conservative fine-tuning of sequence generation models with KL-control »
Natasha Jaques · Shixiang Gu · Dzmitry Bahdanau · Jose Miguel Hernandez-Lobato · Richard E Turner · Douglas Eck -
2017 Talk: Sequence Tutor: Conservative fine-tuning of sequence generation models with KL-control »
Natasha Jaques · Shixiang Gu · Dzmitry Bahdanau · Jose Miguel Hernandez-Lobato · Richard E Turner · Douglas Eck