Towards a Science of AI Agent Reliability

AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a major limitation of current evaluations: focusing on a single metric is not enough to understand agent behavior. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile consisting of twelve metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.

Modern AI systems are increasingly used as "agents". Rather than just answering questions, these systems can take actions on our behalf, such as writing and running code, managing data, or handling customer-service requests. Right now, these agents score ever higher on standard tests, but they still fail in surprising ways once deployed. Examples include (but are not limited to) giving confidently wrong answers or taking damaging actions. We argue that today's evaluations miss much of what makes an agent trustworthy. They mostly ask one question: how often does the agent succeed on average? But a success rate alone ignores whether an agent behaves the same way each time, holds up when the situation changes, recognizes when it is likely to be wrong, and avoids causing serious harm when it does fail. Borrowing principles from fields where reliability is historically important (such as aviation and nuclear power), we build a way to measure four key qualities—consistency, robustness, predictability, and safety—and put this measurement approach to the test on 15 leading AI models. We find that although these models have grown far more capable over the past two years, they have become only slightly more reliable.