Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL.

AI chatbots can be tricked into producing harmful content by cleverly worded "jailbreak" prompts. Keeping them safe is currently a reactive game of whack-a-mole: specialists probe a finished model for weaknesses, engineers patch the holes, and then new attacks appear, so defenses perpetually trail behind the latest threats. We let a model train against itself, a technique called self-play. The same model plays both sides: as an "attacker" it invents new ways to provoke unsafe answers, and as a "defender" it learns to resist them, and both roles improve together. Like sparring partners who keep raising each other's game, neither can coast on old tricks: when one finds a weakness, the other must adapt. We also prove mathematically that if this contest reaches a balance point, the defender is guaranteed to respond safely to any prompt. Models trained this way withstand a broader range of attacks than those trained against a fixed opponent, while remaining just as helpful, and they do so with little extra computing cost. By letting AI systems discover and repair their own vulnerabilities, our method offers a more proactive and more affordable path toward trustworthy AI.