Be a Goldfish: Forgetting Bad Conditioning in Sparse Linear Regression via Variational Autoencoders

Variational Autoencoders (VAEs), a class of latent-variable generative models, have seen extensive use in high-fidelity synthesis tasks, yet their loss landscape remains poorly understood. Prior theoretical works on VAE loss analysis have focused on their latent-space representational capabilities, both in the optimal and limiting cases. Although these insights have guided better VAE designs, they also often restrict VAEs to problem settings where classical algorithms, such as Principal Component Analysis (PCA), can trivially guarantee globally optimal solutions. In this work, we push the boundaries of our understanding of VAEs beyond these traditional regimes to tackle NP-hard sparse inverse problems, for which no classical algorithms exist. Specifically, we examine the nontrivial Sparse Linear Regression (SLR) problem of recovering optimal sparse inputs in the presence of an ill-conditioned design matrix having correlated features. We provably show that, under a linear encoder-decoder architecture incorporating the product of the SLR design matrix with a trainable, sparsity-promoting diagonal matrix, any minimum of VAE loss is guaranteed to be an optimal solution. This property is especially useful for identifying (a) a preconditioning factor that reduces the eigenvalue spread, and (b) the corresponding optimal sparse representation. Lastly, our empirical analysis with different types of design matrices validates these findings and even demonstrates a higher recovery rate at low sparsity where traditional algorithms fail. Overall, this work highlights the flexible nature of the VAE loss, which can be adapted to efficiently solve computationally hard problems under specific constraints.

Suppose you need to purchase a generous bouquet, but the price increases with the choice of more expensive flowers. The difficulty of deciding which flowers to choose to create a visually appealing bouquet within a given budget increases if you are limited to selecting only a few from many. This challenge, also known as sparse recovery, is a hard problem in computer science, for which efficient and reliable solutions do not yet exist.We look at this problem through the lens of generative modeling using machine learning. Their success when applied to text, speech, and image synthesis is due to their ability of learning the hidden properties of the object. However, we demonstrate that they can also be applied to hard, sparse recovery problems by making a sparsity-informed design change to the network architecture.Our proposed generative modeling solution achieves a better sparse recovery rate compared to state-of-the-art algorithms. This is possible because a generative model optimizes for both the number as well as the hidden properties of the features, thereby creating the optimal set of features. Our work opens up the use of generative models for the new problem of sparse recovery, which can even be applied to selecting the flowers for the best-looking bouquet.