Silent Failures in LLM Vulnerability Detection: The Expensive Model Trap

Agents are increasingly used to detect vulnerabilities in code, yet a single run reporting "no findings" can be silently unreliable: the same model may detect a vulnerability on one run and miss it on the next. We construct a benchmark of 50 public CVEs deliberately filtered to exhibit stochastic detection, evaluate 50 LLMs with 10 independent runs each (25,000 trials), and fit a three-dimensional Beta-Binomial multidimensional item response theory model to diagnose the structure of detection failure. Overdispersion is substantial (κ ≈ 6.8): repeated trials vary more than a binomial model predicts. On this benchmark, the strongest single-pass model (expected 26.7/50 CVEs) requires ＄41 in repeated scanning to reach a reliability target of finding ≥45/50 CVEs with ≥80% probability, while a portfolio of cheap models reaches the same target for under ＄1.