FaultLoc: Evaluating Coding Agents For Fault Localization

Coding agents powered by large language models have demonstrated strong performance on security tasks such as vulnerability discovery and automated patching. However, existing benchmarks primarily evaluate patch generation through functional tests, allowing agents to succeed by mitigating symptoms at the crash site without identifying the underlying root cause. As a result, it remains unclear whether these systems perform genuine code reasoning or rely on superficial fixes. We introduce FAULTLOC, a benchmark designed to evaluate fault localization in large codebases. Given a bug report and a proof-of-concept input, agents must identify the code region responsible for the vulnerability, with predictions evaluated against developer patches. FAULTLOC comprises 900 real-world vulnerabilities with verified PoCs and fixes across widely-used open-source projects. Our evaluation shows that even frontier agents fail to recover the developer-identified root-cause region in most cases, achieving only 40% overlap with ground-truth buggy lines and frequently mislocalizing vulnerabilities at the file level. These results highlight the need for agents that reason about causal structure rather than merely suppress observable failures.