SVRG and Beyond via Posterior Correction

Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections. Originally proposed over a decade ago, these methods have never been connected to any Bayesian method at a fundamental level. Here, we fill this gap and derive surprising new connections of SVRG to a recently proposed Bayesian method called `posterior correction'. Our main contribution is to show that SVRG can be recovered as a special case of posterior-correction over isotropic-Gaussian posteriors. Novel extensions of SVRG are automatically obtained by using more flexible exponential-family posteriors. We derive two new such extensions by using Gaussian families: a Newton-like variant with novel Hessian corrections, and an Adam-like extension that scales to large problems. Our work is the first to connect SVRG to Bayes and use it to speed-up training.

Machine learning models are trained by changing their weights in many steps but when these steps are based on only little data, they can be noisy which makes training slow. Earlier methods like SVRG have tried to reduce this noise by looking at all available data every once in a while but found little success for deep learning yet, despite a long history and many adaptations. We show that SVRG can be understood from a different angle as an instance of a Bayesian method called Posterior Correction which gives new insights. From this angle, information from data that has been seen already is corrected by observing new data that is used for the current step. We use this angle to derive existing methods as a special case and new methods that go beyond SVRG. To summarize, our method connects two seemingly unrelated research fields and outlines ways forward to make such techniques work better for deep learning.