RSPO: Regularized Self-Play Alignment of Large Language Models

Self-play-based policy optimization has emerged as an effective approach for fine-tuning large language models (LLMs), formulating preference optimization as a two-player game. However, the regularization with respect to the reference policy, which is crucial for mitigating over-optimization, has been insufficiently investigated in self-play alignment. To study the impact of different regularization strategies, we propose \textbf{Regularized Self-Play Policy Optimization (RSPO)}, a novel framework that unifies prior methods and enables simple plug-and-play regularizers, meanwhile preserving convergence to Nash equilibrium of the corresponding regularized game. We empirically show that RSPO with appropriate regularizers can substantially improve the length-controlled win rate (LCWR) on AlpacaEval-2 across a range of base models, while also achieving consistently superior performance on Arena-Hard, MT-Bench, ArmoRM, and response diversity. In particular, RSPO improves unregularized self-play baseline (SPPO) on AlpacaEval-2 LCWR from $28.5\%$ to $ 35.4\%$ with base model Mistral-7B, from $38.77\%$ to $43.66\%$ with LLaMA-8B, and from $50.54\%$ to $51.83\%$ with Gemma-2B. Combining simplicity, convergence guarantees, and significant empirical gains, RSPO offers a strong foundation for exploring regularized self-play in alignment.

Large language models are increasingly improved by letting them compare and refine their own answers, a process inspired by self-play in games. However, if this process is pushed too far, the model may learn to exploit imperfections in the feedback signal instead of genuinely becoming more helpful. This can lead to unstable training or worse answers. In this paper, we introduce Regularized Self-Play Policy Optimization (RSPO), a simple framework that keeps self-play learning closer to a trusted reference model while still allowing the model to improve. The key idea is to add a flexible “stabilizing” term during training, which helps prevent over-optimization and makes the learning process more reliable. We show that RSPO can be easily added to existing self-play methods and comes with theoretical guarantees. In experiments across several popular benchmarks, RSPO consistently improves the quality of responses from different language models. It also improves response diversity and can achieve strong performance with smaller preference models, making self-play alignment more stable, effective, and practical.