AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition

Recent advances in large language models have enabled LLM-based agents to achieve strong performance on a variety of benchmarks. However, their performance in real-world deployments often that observed on benchmark settings, especially in complex and imperfect environments. This discrepancy largely arises because prevailing training and evaluation paradigms are typically built on idealized assumptions, overlooking the inherent stochasticity and noise present in real-world interactions. To bridge this gap, we introduce AgentNoiseBench, a framework for systematically evaluating the robustness of agentic models under noisy environments. We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios and categorize environmental noise into two primary types: user-noise and tool-noise. Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks while preserving task solvability. Leveraging this pipeline, we perform extensive evaluations across a wide range of models with diverse architectures and parameter scales. Our results reveal consistent performance variations under different noise conditions, highlighting the sensitivity of current agentic models to realistic environmental perturbations.

Artificial intelligence systems, often referred to as agents, are typically built on large language models. While they are becoming highly skilled at solving test problems in controlled, laboratory-like settings, their performance often drops significantly when deployed in messy, unpredictable real-world scenarios. In practice, users may give unclear instructions, and external tools can fail. This performance gap stems from the fact that current methods for testing AI agents are based on ideal, noiseless conditions, which do not reflect the imperfect reality they face. To address this, we created AgentNoiseBench, a new testing platform designed to measure how well AI agents can handle real-world noise and imperfections. Our work identifies and injects two primary types of real-world problems into standard tests: user-noise (which simulates confusing or contradictory instructions) and tool-noise (which simulates receiving incomplete or faulty information from external programs and databases). Through extensive evaluation of diverse state-of-the-art AI agents under these noisy conditions, we demonstrate that they are surprisingly sensitive to such realistic disruptions. This research represents a crucial step toward building more reliable, robust, and trustworthy AI assistants capable of consistent performance outside the laboratory.