Timezone: »

 
Poster
SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization
Yuhta Takida · Takashi Shibuya · WeiHsiang Liao · Chieh-Hsin Lai · Junki Ohmura · Toshimitsu Uesaka · Naoki Murata · Shusuke Takahashi · Toshiyuki Kumakura · Yuki Mitsufuji

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #402
in Poster Session 2 »

One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook, also known as codebook collapse. We hypothesize that the training scheme of VQ-VAE, which involves some carefully designed heuristics, underlies this issue. In this paper, we propose a new training scheme that extends the standard VAE via novel stochastic dequantization and quantization, called stochastically quantized variational autoencoder (SQ-VAE). In SQ-VAE, we observe a trend that the quantization is stochastic at the initial stage of the training but gradually converges toward a deterministic quantization, which we call self-annealing. Our experiments show that SQ-VAE improves codebook utilization without using common heuristics. Furthermore, we empirically show that SQ-VAE is superior to VAE and VQ-VAE in vision- and speech-related tasks.

Keywords: DL: Generative Models and Autoencoders · OPT: Stochastic · PM: Bayesian Models and Methods · T: Probabilistic Methods · MISC: Representation Learning

Author Information

Yuhta Takida (Sony Group Corporation)
Takashi Shibuya (Sony Group Corporation)
WeiHsiang Liao (Sony Group Corporation)
Chieh-Hsin Lai (Sony Group Corporation)
Junki Ohmura (Sony Group Corporation)
Toshimitsu Uesaka (Sony Group Corporation)
Naoki Murata (Sony Group Corporation)
Shusuke Takahashi (Sony Group Corporation)
Toshiyuki Kumakura (Sony Corporation of America)
Yuki Mitsufuji (Sony Group Corporation)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors