MAGIC: A Co-Evolving Attacker–Defender Adversarial Game for Robust LLM Safety

Ensuring robust safety alignment is crucial for Large Language Models (LLMs), yet existing defenses often lag behind evolving adversarial attacks due to their \textbf{reliance on static, pre-collected data distributions}. In this paper, we introduce \textbf{MAGIC}, a novel multi-turn multi-agent reinforcement learning framework that formulates LLM safety alignment as an adversarial asymmetric game. Specifically, an attacker agent learns to iteratively rewrite original queries into deceptive prompts, while a defender agent simultaneously optimizes its policy to recognize and refuse such inputs. This dynamic process triggers a \textbf{co-evolution}, where the attacker's ever-changing strategies continuously uncover long-tail vulnerabilities, driving the defender to generalize to unseen attack patterns. Remarkably, we observe that the attacker, endowed with initial reasoning ability, evolves \textbf{novel, previously unseen combinatorial strategies} through iterative RL training, underscoring our method’s substantial potential. Theoretically, we provide insights into a more robust game equilibrium and derive safety guarantees. Extensive experiments validate our framework's effectiveness, demonstrating superior defense success rates without compromising the helpfulness of the model.

Large language models are increasingly used in real-world applications, but they can still be manipulated into producing unsafe or harmful responses. Existing safety methods often rely on fixed examples of known attacks, which makes them less effective when attackers invent new ways to bypass safeguards. This paper introduces MAGIC, a training framework that treats language model safety as an iterative game between two models: an attacker and a defender. The attacker tries to rewrite harmful requests in more deceptive ways, while the defender learns to recognize and refuse these requests without refusing normal, harmless user queries. Through repeated interaction, both sides improve: the attacker discovers new weaknesses, and the defender becomes more robust against them. Our experiments show that this process improves the model's ability to resist harmful prompts while largely preserving its usefulness on benign tasks. We also find that the attacker can develop new combinations of attack strategies during training, suggesting that safety evaluation should account for adaptive and evolving threats rather than only fixed test cases.