AgentSuite: Toward More Reliable Agent Evaluation with a Component-Based Benchmark Auditing Pipeline

Reliable evaluation of large language model (LLM) agents depends critically on benchmark validity, yet modern agent benchmarks often contain hidden flaws arising from interactions among user instructions, environments, tools, ground-truth trajectories, and evaluation protocols. These flaws confound model errors with benchmark artifacts and undermine leaderboard-based comparisons. We propose **COBA** (**CO**mponent-based **B**enchmark **A**uditing), an automated pipeline for diagnosing and filtering validity issues in agent benchmarks. COBA decomposes agent tasks into four standardized components—User, Environment, Ground Truth, and Evaluation—and operationalizes a component-level issue taxonomy using hybrid rule-based detectors and taxonomy-guided LLM evaluation. Across six widely used agent benchmarks, COBA achieves strong alignment with expert judgments, with F1 scores between 0.791 and 0.874. It complements manual verification of $\tau^2$-bench by identifying issues missed due to benchmark complexity, and generalizes to previously unseen benchmarks with minimal adaptation. Our analysis shows that benchmark flaws are widespread and materially affect evaluation outcomes, demonstrating that component-based auditing provides a scalable foundation for more reliable and interpretable agent evaluation. We release AgentSuite, a unified benchmark-running platform that includes the COBA auditing pipeline and audited benchmark variants: https://github.com/Agent-Suite/AgentSuite.

Large language model (LLM) agents can use tools, follow instructions, and interact with environments to complete tasks. Researchers compare these agents using benchmarks, but the benchmarks themselves can contain hidden mistakes, such as unclear instructions, wrong tool results, incorrect expected answers, or unfair scoring. These problems make it hard to tell whether an agent truly failed or whether the test was flawed. We introduce COBA (COmponent-based Benchmark Auditing), an automated pipeline for auditing agent benchmarks. COBA is based on the insight that many benchmark errors arise from mismatches between connected parts of a task: the user request, the tools and environment, the expected answer or actions, and the scoring rules. We first organize common benchmark problems into a component-based taxonomy, then use rule-based checks and LLM-assisted review to detect them across the connected parts of each task. Across widely used agent benchmarks, COBA closely matches expert human reviewers and finds issues that are difficult to catch manually. We also show that flawed tasks can change model evaluation outcomes, meaning benchmark quality directly affects how researchers judge progress in agents. We release AgentSuite, a unified platform that includes COBA, benchmark-running tools, and audited benchmark materials to support more reliable and reproducible agent evaluation.