Efficient Stochastic Optimisation via Sequential Monte Carlo

The problem of optimising functions with intractable gradients frequently arises in machine learning and statistics, ranging from maximum marginal likelihood estimation procedures to fine-tuning of generative models. Stochastic approximation methods for this class of problems typically require inner sampling loops to obtain (biased) stochastic gradient estimates, which rapidly becomes computationally expensive. In this work, we develop sequential Monte Carlo (SMC) samplers for optimisation of functions with intractable gradients. Our approach replaces expensive inner sampling methods with efficient SMC approximations, which can result in significant computational gains. We establish convergence results for the basic recursions defined by our methodology which SMC samplers approximate. We demonstrate the effectiveness of our approach on the reward-tuning of energy-based models within various settings.

Many modern machine learning methods learn by repeatedly adjusting model parameters in the direction that improves performance. For many important models, however, this direction cannot be computed directly. Instead, it must be estimated by sampling from a probability model that changes after every parameter update. This can make learning slow, since every step requires solving a new sampling problem. This paper introduces SOSMC, which leverages Sequential Monte Carlo, a particle-based method for tracking changing probability models. Instead of starting the sampling process from scratch at every step, SOSMC maintains a collection of representative samples, called particles, that evolve alongside the model. After each update, the particles are moved and reweighted so that they remain representative of the new probability model, and are then used to estimate the next update direction. The result is a more efficient way to learn in models where exact update directions are unavailable. Experiments in reward tuning and Bayesian image deblurring show improved reliability and efficiency. Understanding this behaviour fully for finite numbers of particles remains an important direction for future work.