Bias and Variance in Value Function Estimation
Shie Mannor - MIT Duncan Simester - MIT Peng Sun - Duke John Tsitsiklis - MIT
We consider the bias and variance of value function estimation that are caused by using an empirical model instead of the true model. We analyze these bias and variance for Markov processes from a classical (frequentist) statistical point of view, and in a Bayesiansetting. Using a second order approximation, we provide explicit expressionsfor the bias and variance in terms of the transition counts and the rewardstatistics. We present supporting experiments with artificial Markov chains and with a large transactional database provided by a mail-order catalog firm.