Offline Two-Player Zero-Sum Markov Games with KL Regularization

We study the problem of learning Nash equilibria in offline two-player zero-sum Markov games. While existing approaches often rely on explicit pessimism to address distribution shift, we show that KL regularization alone suffices to stabilize learning and guarantee convergence. We first introduce Regularized Offline Sequential Equilibrium (ROSE), a theoretical framework that achieves a fast $ \widetilde{\mathcal{O}}(1/n) $ convergence rate under *unilateral concentrability*, improving over the standard $ \widetilde{\mathcal{O}}(1/\sqrt{n}) $ rates in unregularized settings. We then propose Sequential Offline Self-play Mirror Descent (SOS-MD), a practical model-free algorithm based on least-squares value estimation and iterative self-play updates. We prove that the last iterate of SOS-MD attains the same $ \widetilde{\mathcal{O}}(1/n) $ statistical rate up to a vanishing optimization error of order $ \widetilde{\mathcal{O}}(1/\sqrt{T}) $ in the number of self-play iterations $T$.

We study offline learning for two-player adversarial games — settings where an agent must perform well against an opponent, using only a fixed dataset of past interactions rather than further experimentation. This setting arises when collecting new data is too expensive, slow, or dangerous. Standard methods handle the gaps in such datasets by adding hand-crafted penalties to discourage uncertain decisions, a strategy known as pessimism. We instead use KL regularization — a technique that keeps the learned behavior close to a reference policy, also used in modern language model training. We show that this regularization alone suffices to stabilize offline learning, eliminating the need for pessimism, and we establish a faster learning rate than was known for prior offline approaches. We further present a practical algorithm that approximates the same procedure efficiently, with the same statistical guarantees.