MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. Inspired by Stackelberg security games, we formalize this problem as a game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary–Meta-Agent (MaMa), a novel algorithm inspired by this formalization for automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting. Code is available at https://github.com/JNoether/MaMa

We explored systems in which multiple AI agents work together to solve a task. We investigated what would happen if one of these agents misbehaved by executing dangerous actions instead of solving the task, such as sending spam emails, purchasing unrelated items, or deleting important files on the user's device. To study this, we set up a game between two players: a system designer, which proposes how the agents should be organized and interact, and an attacker, which tries to find the best way to compromise one of the agents to cause harm. The designer then uses the attacker's discoveries to reorganize the system, for example by adding oversight agents that monitor and filter the actions of others, making it harder for any single compromised agent to cause damage. We found that simple systems are unprepared for misbehaving agents, but that iterating through this game yields systems that are both significantly safer and equally capable of solving their assigned tasks. We believe this work will help practitioners design systems that remain safe even in worst-case scenarios.