The Leaderboard Lottery: Why Top LLM Rankings Are Often Coin Flips, and How to Fix Them

LLM leaderboards rank models by point-estimate accuracy, but the claim users care about—"is the top-ranked model truly best?"—is a post-selection event across all N models. Marginal per-model confidence intervals do not cover this event. On benchmark data calibrated to the Open LLM Leaderboard v2, the reported winner is the population-level best only 0–58% of the time under prompt resampling, far below the nominal 95%. We adapt paired-bootstrap Multiple Comparisons with the Best (MCB) to return a family-wise-error-controlled confidence set of plausible winners. Across synthetic data and six calibrated leaderboard tasks, MCB restores ≥95% coverage while revealing that many benchmarks cannot statistically distinguish their top models: the 95% sets contain 8–35 models. A simulated Chatbot Arena study shows the same post-selection instability for pairwise preference rankings. The correction is simple, requires only per-prompt scores, and can be adopted by leaderboard maintainers immediately.