Generative Modeling of Discrete Latent Structures via Dynamic Policy Gradients

Many scientific problems require inferring unobserved mechanistic latent states from indirect observations. While classical approaches, including expectation-maximization, do not scale to combinatorially large spaces, deep learning approaches such as variational autoencoders typically form artificial latent states rather than reconstructing the mechanistic ground-truth states. Here, we introduce GReinSS, a policy learning framework that uses dynamically rescaled rewards to learn latent state distributions that maximize the observed data likelihood. We show that GReinSS accurately reconstructs simulated latent sets and latent graphs, outperforming alternative policy learning and generative modeling baselines. Additionally, GReinSS reconstructs isoforms from real short-read RNA sequencing data that better match orthogonal long-read sequencing detected isoforms than the standard RSEM algorithm. Overall, GReinSS is a principled and practically effective approach for generative modeling and inference of combinatorial latent states from indirect observations.

Many real-world problems require inferring hidden underlying states from indirect and noisy observations. Classical approaches such as expectation maximization often struggle when the number of possible hidden states becomes extremely large. In addition, modern deep learning methods such as variational autoencoders typically form artificial latent states rather than reconstructing the mechanistic ground-truth states. In this work, we introduce GReinSS, a machine learning framework for reconstructing combinatorial hidden states from indirect observations. GReinSS combines ideas from probabilistic modeling and reinforcement learning to efficiently search through large spaces of possible hidden states while directly optimizing how well the inferred states explain the observed data. Across several simulated problems, GReinSS more accurately reconstructs hidden sets and graphs than existing generative modeling and reinforcement learning approaches. We also apply GReinSS to short read RNA sequencing data and show that it reconstructs RNA isoforms that more closely match independent long-read sequencing measurements than the standard RSEM method. Overall, GReinSS provides a principled and broadly applicable framework for inferring complex hidden structures from indirect observations.