A Game-Theoretic Analysis of Attacks on Large Language Models via Compositional Skills

As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker hiding its intent via compositional skills and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks.

Large language models are trained to refuse harmful requests, such as instructions for cyberattacks or dangerous activities. However, people can still sometimes bypass these safeguards using carefully crafted prompts that disguise harmful intentions inside seemingly harmless language. We wanted to better understand why these attacks are so effective and why defending against them remains difficult even for advanced AI systems. We studied this problem by modeling the interaction between attackers and defenders as a strategic game. In our framework, attackers try to hide harmful goals by combining different skills such as role-playing, metaphors, fictional stories, or hypothetical scenarios. For example, instead of directly asking for harmful information, an attacker may hide the request inside a poem or fictional conversation. We found that many existing methods can be understood as variations of this same strategy. Our analysis also revealed an important challenge: attackers can often create many more possible attack combinations than defenders can realistically monitor. These findings suggest important limits of straightforward defense approaches and have implications for designing more effective defense strategies.