DRIFT-BENCH: Diagnosing CoopeRative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction

As Large Language Models transition to autonomous agents, user inputs frequently violate cooperative assumptions (e.g., implicit intent, missing parameters, false presuppositions, or ambiguous expressions), creating execution risks that text-only evaluations do not capture. Existing benchmarks typically assume well-specified instructions or restrict evaluation to text-only, single-turn clarification, and thus do not measure multi-turn disambiguation under grounded execution risk. We introduce DRIFT-BENCH, the first diagnostic benchmark that evaluates agentic pragmatics under input faults through multi-turn clarification across state-oriented and service-oriented execution environments. Grounded in classical theories of communication, DRIFT-BENCH provides a unified taxonomy of cooperative breakdowns and employs a persona-driven user simulator with the Rise evaluation protocol. Experiments show substantial performance drops under these faults, with clarification effectiveness varying across user personas and fault types. DRIFT-BENCH connects clarification studies with agent benchmarking, providing a framework to diagnose failures arising from faulty user inputs.

As AI systems become agents that can use tools, call services, edit files, or change digital environments, unclear user instructions can lead to real mistakes. People often give requests that are incomplete, ambiguous, based on a wrong assumption, or mixed with unrelated information. Today’s evaluations usually assume that users always give clear and correct instructions, which misses an important safety problem. This paper introduces DRIFT-BENCH, a test suite for studying how AI agents handle flawed user requests through conversation. The benchmark checks whether agents notice when something is unclear, ask useful follow-up questions, and avoid acting too quickly when an action may be risky. We test agents in both transparent environments, where they can inspect the current state, and service-like environments, where they must rely on tool responses. Our results show that current agents often perform much worse when user inputs are flawed, and they sometimes take risky actions instead of asking for clarification. This highlights the need for agents that are better at careful, cooperative communication before acting.