Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over-corrects, introducing a bias toward longer answers instead. We first analyze these scoring rules, characterizing when standard and length-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths. Motivated by this analysis, we introduce Bayesian accuracy, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects. Bayesian accuracy is a drop-in replacement for likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.

The paper is about a problem in multiple-choice language model tests: longer answers often get lower probability scores just because they contain more tokens. The usual fix is to divide by answer length, but the paper shows this can overcorrect and start unfairly favoring longer answers instead. The proposed solution, Bayesian accuracy, estimates how much answer length is affecting the score and subtracts that effect directly. It is meant to be a simple drop-in replacement for standard scoring: no extra model runs are needed, and in the experiments it reduces length bias more reliably than common metrics and other proposed fixes.