When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems

While enabling effective collaboration on complex tasks, LLM-based Multi-Agent Systems (MAS) face critical security challenges due to vulnerabilities at the agent and interaction levels. Most existing MAS security defenses are built upon two core assumptions: semantically-explicit malicious attacks and explicit graph-based modeling of the MAS topology and agent-level interactions. In practice, real-world attacks are becoming more semantically stealthy, while MAS execution is typically asynchronous without the temporal alignment assumed by graph-based propagation models. To address these limitations, we propose AcMAS, an activation-based framework for malicious-behavior detection in MAS. By analyzing internal reasoning states in the activation space of local agents, AcMAS detects even stealthy attacks in a synchronization-robust fashion, without relying on explicit interaction graphs. Moreover, our activation analysis provides critical signals to guide AcMAS in restoring the functionality of compromised agents, rather than the disruptive agent isolation commonly used by the state-of-the-art methods. Comprehensive evaluation demonstrates that AcMAS significantly outperforms graph-based baselines against stealthy attacks, by +0.22 F1 in synchronous settings (0.94 vs. 0.72) and by +0.55 F1 in asynchronous settings (0.93 vs. 0.38), with generalization across diverse open-source LLM backbones, attack intensity, and MAS scale.

AI systems increasingly rely on teams of language model "agents" that collaborate to solve complex tasks, but a single compromised agent can corrupt the entire team. Existing defenses look at what agents say to each other, which fails when attacks are subtle or when agents don't take turns. We introduce AcMAS, which instead inspects what agents are thinking by analyzing their internal neural activations, catching stealthy attacks that hide in normal-looking messages. Across five attack scenarios and multiple AI models, AcMAS detects malicious agents with 92–95% accuracy, far surpassing prior methods (around 56–72%). Unlike defenses that simply remove suspicious agents and break the team, AcMAS gently steers compromised agents back to healthy reasoning, preserving 97% of task completion while blocking nearly all attacks. The result: safer, self-healing AI collaborations.