Variational Sparse Coding with Learned Thresholding

Kion Fallah · Christopher Rozell

Hall E #721

Keywords: [ MISC: Unsupervised and Semi-supervised Learning ] [ DL: Generative Models and Autoencoders ] [ PM: Bayesian Models and Methods ]

[ Abstract ]
[ Poster [ Paper PDF
Wed 20 Jul 3:30 p.m. PDT — 5:30 p.m. PDT
Spotlight presentation: PM: Variational Inference/Bayesian Models and Methods
Wed 20 Jul 7:30 a.m. PDT — 9 a.m. PDT


Sparse coding strategies have been lauded for their parsimonious representations of data that leverage low dimensional structure. However, inference of these codes typically relies on an optimization procedure with poor computational scaling in high-dimensional problems. For example, sparse inference in the representations learned in the high-dimensional intermediary layers of deep neural networks (DNNs) requires an iterative minimization to be performed at each training step. As such, recent, quick methods in variational inference have been proposed to infer sparse codes by learning a distribution over the codes with a DNN. In this work, we propose a new approach to variational sparse coding that allows us to learn sparse distributions by thresholding samples, avoiding the use of problematic relaxations. We first evaluate and analyze our method by training a linear generator, showing that it has superior performance, statistical efficiency, and gradient estimation compared to other sparse distributions. We then compare to a standard variational autoencoder using a DNN generator on the CelebA dataset.

Chat is not available.