Safety Alignment of LMs via Non-cooperative Games

Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models. Code at github.com/facebookresearch/advgame.

Language models like ChatGPT can sometimes be tricked into producing harmful content through clever prompts called "jailbreaks." The standard fix is to collect known bad prompts and train the model to refuse them, but this is a static defense that falls behind as new attacks appear. We treat safety training as a game between two language models. One model, the Attacker, learns to find new ways to elicit harmful behavior. The other, the Defender, learns to respond helpfully without producing harm. The two are trained together, each adapting to the other, so the Defender always faces fresh, evolving attacks instead of a fixed set. The result is two useful models. The Defender ends up more resistant to jailbreaks while staying just as helpful on normal questions. The Attacker becomes a general-purpose red-teaming tool that can test other models for safety gaps. Compared to existing methods, our approach reaches a better balance between safety and usefulness.