TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation

Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either constrain models to a compliance trap by treating existing code as the ground truth for regression prevention, or rely on post-failure artifacts (e.g., issue reports) for reactive bug reproduction, failing to expose defects before they manifest as failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. Comprising 2,389 tasks across 482 repositories, TestExplora conceals all defect-related information, forcing models to uncover bugs by identifying discrepancies between implementation and documentation-derived intent—utilizing documentation as the reference oracle. Furthermore, to ensure sustainable evaluation and mitigate risks of data leakage in static datasets, we propose a continuous, time-aware data collection framework. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass ($F2P$) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an $F2P$ of 17.27% and an $F2P@5$ of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.

Problem. Software bugs that reach real users can cause serious damage—lost data, outages, and costly failures. People are starting to use AI models to automatically write tests that catch bugs, but today's benchmarks only check whether LLM can confirm code already works or recreate bugs humans have already found. What we actually want is that LLM can spot unknown bugs hiding in a codebase before they cause harm. Solution. We built TestExplora, a benchmark with 2,389 testing tasks from 482 real GitHub projects. Instead of telling the models where the bug is, we give it only the documentation describing what the code is supposed to do, and ask it to write tests that reveal where the code falls short. A test only counts as successful if it catches the real bug. We also designed the benchmark to refresh itself with newly fixed bugs over time, so models can't simply memorize the answers.