ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox

Current LLM agents are proficient at calling isolated APIs but struggle with the "last mile" of commercial software automation. In real-world scenarios, tools are not independent; they are atomic, interdependent, and prone to environmental noise. We introduce $\textbf{ComplexMCP}$, a benchmark designed to evaluate agents in these rigorous conditions. Built on the Model Context Protocol (MCP), $\textbf{ComplexMCP}$ provides over 300 meticulously tested tools derived from 7 stateful sandboxes, ranging from office suites to financial systems. Unlike existing datasets, our benchmark utilizes a seed-driven architecture to simulate dynamic environment states and unpredictable API failures, ensuring a deterministic yet diverse evaluation. We evaluate various LLMs across full-context and RAG paradigms, revealing a stark performance gap: even top-tier models fail to exceed a 60% success rate, far trailing human performance 90%. Granular trajectory analysis identifies three fundamental bottlenecks: (1) $\textbf{tool retrieval saturation}$ as action spaces scale; (2) $\textbf{over-confidence}$, where agents skip essential environment verifications; and (3) $\textbf{strategic defeatism}$, a tendency to rationalize failure rather than pursuing recovery. These findings underscore the insufficiency of current agents for interdependent workflows, positioning $\textbf{ComplexMCP}$ as a critical testbed for the next generation of resilient autonomous systems. The codebase and benchmark implementation are publicly available at https://github.com/AIDC-AI/complex-mcp.

Large language model agents are becoming good at using single tools, but real software tasks are much harder because tools depend on each other, the environment changes over time, and errors can happen unexpectedly. To study this more realistic setting, we built ComplexMCP, a benchmark that tests AI agents in large software-like environments with over 300 tools spanning messaging, shopping, flights, finance, weather, and news. Our benchmark creates different but repeatable task situations using random seeds, so agents face varied environments and occasional tool failures while experiments remain scientifically reproducible. We also designed an automatic scoring system that checks whether the agent changed the software state correctly, rather than relying on subjective judgments. When we tested many leading AI models, even the best system succeeded on only about 55% of tasks, far below human performance above 90%. Our analysis shows three main weaknesses: models struggle when too many tools are available, often assume the environment is in a clean default state without checking it, and tend to give up too early when recoverable errors occur. These results show that today’s AI agents are still not reliable enough for complex software automation, and ComplexMCP offers a practical testbed for building more robust and trustworthy agents.