Can LLMs Detect Benchmark Defects? A Meta-Benchmark from Benchmark Updates

Every major coding benchmark has released updated versions acknowledging substantial defect rates, yet quality assurance remains largely manual. We propose Task Verification Bench, a meta-benchmark that repurposes author-acknowledged defects from benchmark updates as ground truth and evaluates whether LLM agents can detect them given only the original artifacts. Because a defect diagnosis must be actionable to guide a fix, we evaluate with root-cause matching - requiring the agent to identify the correct defect, not just flag one - which reveals that verdict-level recall overestimates true detection by up to 54pp. A GPT-5.4+Codex verify-only baseline achieves 35~60% root-cause-matched recall across SWE-bench and two non-overlapping splits of Terminal-Bench - meaningful for triage, but far from fully automated auditing. Pipeline ablation uncovers self-attribution anchoring in GPT-5.4 - harness access causes the agent to blame its own code rather than the benchmark - while Gemini tends to resist this effect, showing that model choice matters more than pipeline design. These results suggest that LLM-assisted auditing can meaningfully reduce human review effort for text-layer defects, while infrastructure-layer verification remains a challenge requiring human expertise.