CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting

Publishing a large language model (LLM) benchmark (especially its ground-truth answers) on the Internet risks contaminating future LLMs and enabling evaluation gaming: it may be unintentionally (or intentionally) used to train or select a model, or exploited to overfit and hack leaderboards when labels are accessible. A common mitigation is to keep the benchmark private and let participants submit their models or predictions to the organizers, but this still permits test-set overfitting through feedback loops. To overcome this issue, we propose CapBencher, a way to publish benchmarks without fully disclosing the ground-truth answers, while preserving open evaluation of LLMs. The main idea is to reduce the best possible accuracy, i.e., Bayes accuracy, by injecting randomness to the answers by preparing several logically correct answers, and only include one of them as the solution in the benchmark. Not only does this obscure the ground-truth answers, but it also offers a test for leakage or gaming: since even fully capable models should not surpass the Bayes accuracy, any model that does is a strong signal. We show theoretically and empirically that CapBencher accurately detects test-set overfitting across diverse benchmarks, models, training methodologies, and scenarios.

Language models are judged using benchmarks, which are large sets of questions with known correct answers. But benchmarks are usually posted publicly, and their answers can leak into the data used to train new models. A model can then score well simply because it has effectively seen the answer key beforehand, much like a student who memorized a leaked exam, so the score no longer reflects real ability. Building good benchmarks takes enormous effort, yet they can lose their value soon after release. We propose a way to publish benchmarks with a built-in alarm for this problem. For each question, instead of revealing the single true answer, we accept several equally correct answers and publish just one of them, chosen at random. This caps the highest score any honest model could reach. Since no genuinely capable model should exceed this ceiling, one that does is almost certainly relying on leaked answers, a clear, statistically testable warning sign. The modified benchmark still ranks models in the same order, so it remains useful for measuring progress. This gives benchmark creators a way to keep their tests trustworthy after public release and to catch inflated scores, without any access to the models' inner workings.