SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80\%. However, we reveal that this performance is inflated: our re-evaluation demonstrates that one in five ``solved'' patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation utilizing program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible-but-incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2\% of instances (a $25.1\times$ improvement over prior work) and rejects 19.78\% of previously passing patches. Consequently, the top agent's score decreases from 78.80\% to 62.20\%, causing significant leaderboard reshuffling (e.g., the top-ranked agent drops to 5th place).

Artificial Intelligence (AI) has become remarkably good at automatically fixing software bugs. On a popular industry leaderboard, the top AI models appear to solve nearly 80% of coding problems, suggesting they are almost ready to handle routine programming tasks independently. However, our study reveals these high scores are inflated. We discovered that one out of every five AI-generated solutions is actually incorrect; they only "passed" because the existing exams were too weak to catch their mistakes. To address this, we developed a stricter testing framework called SWE-ABS. It upgrades the evaluation system by uncovering hidden, untested areas in the code and intentionally creating realistic mistakes to see if the AI can spot them. With our tougher grading system, the leading AI's success rate plummeted by over 15%, causing a major leaderboard reshuffle. This work provides a crucial tool to ensure AI-generated software is truly safe before real-world use.