Timezone: »

 
Poster
Variance Reduction in Stochastic Particle-Optimization Sampling
Jianyi Zhang · Yang Zhao · Changyou Chen

Tue Jul 14 09:00 AM -- 09:45 AM & Wed Jul 15 08:00 PM -- 08:45 PM (PDT) @
in Poster Session 3 »
Stochastic particle-optimization sampling (SPOS) is a recently-developed scalable Bayesian sampling framework unifying stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD) algorithms based on Wasserstein gradient flows. With a rigorous non-asymptotic convergence theory developed, SPOS can avoid the particle-collapsing pitfall of SVGD. However, the variance-reduction effect in SPOS has not been clear. In this paper, we address this gap by presenting several variance-reduction techniques for SPOS. Specifically, we propose three variants of variance-reduced SPOS, called SAGA particle-optimization sampling (SAGA-POS), SVRG particle-optimization sampling (SVRG-POS) and a variant of SVRG-POS which avoids full gradient computations, denoted as SVRG-POS$^+$. Importantly, we provide non-asymptotic convergence guarantees for these algorithms in terms of the 2-Wasserstein metric and analyze their complexities. The results show our algorithms yield better convergence rates than existing variance-reduced variants of stochastic Langevin dynamics, though more space is required to store the particles in training. Our theory aligns well with experimental results on both synthetic and real datasets.
Keywords: Bayesian Deep Learning · Bayesian Methods · Bayesian Nonparametrics · Deep Learning - General

Author Information

Jianyi Zhang (Duke University)
Yang Zhao (University at Buffalo)
Changyou Chen (SUNY Buffalo)

More from the Same Authors