AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

LLM agents are increasingly deployed in long-horizon, complex environments to solve challenging problems, but this expansion exposes them to long-horizon attacks that exploit multi-turn user–agent–environment interactions to achieve objectives infeasible in single-turn settings. To measure agent vulnerabilities to such risks, we present AgentLAB, the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks. Currently, AgentLAB supports five novel attack types including intent hijacking, tool chaining, task injection, objective drifting, and memory poisoning, spanning 28 realistic agentic environments, and 644 security test cases. Leveraging AgentLAB, we evaluate representative LLM agents and find that they remain highly susceptible to long-horizon attacks; moreover, defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats. We anticipate that AgentLAB will serve as a valuable benchmark for tracking progress on securing LLM agents in practical settings. The code for AgentLAB is available at: https://tanqiujiang.github.io/AgentLAB_main.

AI assistants are increasingly being given the ability to take actions on behalf of users, such as sending emails, managing calendars, or browsing the web. While much attention has been paid to preventing these systems from being tricked by a single malicious instruction, real-world attackers can be far more patient, gradually steering an AI assistant's behavior over the course of a long conversation or through a series of seemingly harmless interactions. We created AgentLAB, a testing framework that measures how vulnerable AI assistants are to these slow, multi-step attacks. AgentLAB includes five types of attacks across diverse task environments and various test scenarios. When we tested popular AI systems, we found that most remain highly vulnerable to these gradual attacks, even when they successfully resist simpler, one-shot tricks. We also found that existing safety measures designed for single interactions do not reliably protect against these longer, more patient strategies. We hope AgentLAB will help the research community track progress in making AI assistants more resistant to these real-world threats.