Abstract

Variational methods for the Dirichlet process
David Blei - U. C. Berkeley Michael Jordan - U. C. Berkeley
Variational inference methods, including mean field methods and loopy beliefpropagation, have been widely used for approximate probabilistic inference ingraphical models. While often less accurate than MCMC, variational methodsprovide a fast deterministic approximation to marginal and conditionalprobabilities. Such approximations can be particularly useful in highdimensional problems where sampling methods are too slow to be effective. Alimitation of current methods, however, is that they are restricted toparametric probabilistic models. MCMC does not have such a limitation;indeed, MCMC samplers have been developed for the Dirichlet process (DP), anonparametric distribution on distributions (Ferguson, 1973) that is thecornerstone of Bayesian nonparametric statistics~\citep (Escobar and West,1995; Neal, 2000}. In this paper, we develop a mean-field variationalapproach to approximate inference for the Dirichlet process, where theapproximate posterior is based on the truncated stick-breaking construction(Ishwaran and James, 2001). We compare our approach to DP samplers forGaussian DP mixture models.

Variational methods for the Dirichlet process

David Blei - U. C. Berkeley
Michael Jordan - U. C. Berkeley

Variational inference methods, including mean field methods and loopy beliefpropagation, have been widely used for approximate probabilistic inference ingraphical models. While often less accurate than MCMC, variational methodsprovide a fast deterministic approximation to marginal and conditionalprobabilities. Such approximations can be particularly useful in highdimensional problems where sampling methods are too slow to be effective. Alimitation of current methods, however, is that they are restricted toparametric probabilistic models. MCMC does not have such a limitation;indeed, MCMC samplers have been developed for the Dirichlet process (DP), anonparametric distribution on distributions (Ferguson, 1973) that is thecornerstone of Bayesian nonparametric statistics~\citep (Escobar and West,1995; Neal, 2000}. In this paper, we develop a mean-field variationalapproach to approximate inference for the Dirichlet process, where theapproximate posterior is based on the truncated stick-breaking construction(Ishwaran and James, 2001). We compare our approach to DP samplers forGaussian DP mixture models.