Spectral Perturbation Bounds for Experience Replay: A Bias–Variance Decomposition for Offline Decision-Making

Offline decision-making relies on datasets collected from heterogeneous and often suboptimal policies, leading to a fundamental trade-off between statistical efficiency and distribution shift. We study this trade-off by modeling an offline dataset as inducing a mixture Markov kernel over state transitions. We show that the diversity of the dataset improves statistical efficiency by accelerating mixing, while deviations between behavior policies introduce bias that scales with policy distance. Our analysis yields a bias-variance decomposition in which both terms are governed by the environment's mixing time. This provides a principled explanation for conservative methods in offline RL and predicts that the value of diverse datasets is strongly environment-dependent. We theoretically formalize this spectral framework and provide empirical validations using synthetic MDPs to demonstrate the tight coupling between spectral perturbations and the efficacy of offline dataset utilization.