Position: Benchmark Method-Comparisons Are Posterior Identifiability Problems

The dominant evaluation paradigm in foundation-model and machine-learning systems research compares methods by their headline scores on a small handful of benchmarks. We argue that this paradigm is, structurally, a posterior identifiability problem: the observed score gap is a function of multiple latent contributions (algorithmic effect, workload-shape interaction, hardware-implementation interaction), and the posterior over the algorithmic component is identifiable only when the experimental design provides enough linearly independent observations. We formalize the problem as a linear-Gaussian inverse problem with rank-based identifiability and prove a posterior identifiability proposition with a closed-form posterior variance expression. Two empirical experiments demonstrate the implications: a phase-transition experiment showing posterior std on the algorithmic effect contracts from 0.75 (essentially the prior) at one workload to 0.012 at five workloads, with the transition occurring at three workloads (the rank of the parameter space); and a condition-number sweep showing posterior contraction degrades smoothly from 95% (well-conditioned) to 27% (near-singular) at fixed rank. We propose that benchmark-comparison papers report posterior identifiability diagnostics: latent-parameter inventory, design-matrix rank, posterior contraction relative to prior, and posterior credible intervals on the headline gap.