Timezone: »

 
Poster
Batch Stationary Distribution Estimation
Junfeng Wen · Bo Dai · Lihong Li · Dale Schuurmans

Wed Jul 15 10:00 AM -- 10:45 AM & Wed Jul 15 09:00 PM -- 09:45 PM (PDT) @
in Poster Session 24 »

We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a \emph{fixed} set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.

Keywords: Monte Carlo Methods · Reinforcement Learning · Probabilistic Inference - Approximate, Monte Carlo, and Spectral Methods

Author Information

Junfeng Wen (University of Alberta)
Bo Dai (Google Brain)
Lihong Li (Google Research)
Dale Schuurmans (University of Alberta)

More from the Same Authors