A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As consequences, we complement classical results with a covariance dynamics, a local Ornstein-Uhlenbeck description, an explicit estimate on the mixing time influence on convergence, and a new range of admissible stepsizes.