Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games

Two-player games such as board games have long been used as traditional benchmarks for reinforcement learning. This work revisits a policy optimization method with reverse Kullback-Leibler regularization and entropy regularization and analyzes this combination in two-player zero-sum settings from theoretical and empirical perspectives. From a theoretical perspective, we investigate the stability of the policy update rule in two theoretical settings: game-theoretic normal-form games and finite-length games. We provide novel convergence guarantees and verify our theoretical results through numerical experiments on synthetic games. From an empirical perspective, we derive a practical model-free reinforcement learning algorithm based on the regularized policy optimization. We validate the training efficiency of our algorithm through comprehensive experiments on five board games: Animal Shogi, Gardner Chess, Go, Hex, and Othello. Experimental results show that our agent learns more efficiently than existing methods across environments.

Strong game-playing AI systems based on reinforcement learning, such as AlphaZero, have achieved remarkable success by using look-ahead search to choose better moves. While this approach has been highly successful, it requires a large number of simulations during training, making it computationally expensive. As a result, building or reproducing strong game-playing AI has often been limited to researchers with enormous computing resources. In this work, we propose a method that learns good moves directly from self-play, without using look-ahead search during training. By keeping its strategy from changing too abruptly as self-play progresses, our method stabilizes learning even without search. Experiments show that it can reach comparable performance to existing search-based methods using less computation. Reducing the computation needed for training brings game-playing AI closer to a technology that more researchers and practitioners can use. From the standpoint of scientific reproducibility, lightweight methods also make experiments easier to verify. Moreover, our approach can serve as a promising baseline for applying reinforcement learning to new environments and decision-making problems, providing a foundation for further technical innovation.