When are likely answers right? On Sequence Probability and Correctness in LLMs

Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we quantify this relationship across various decoding methods, models, and benchmarks. Surprisingly, we find that such a relationship often exists, but depends on the task, model, and decoding method. In particular, sequence-level methods that target the power distribution often generate sequences that are both more likely and more correct than a low-temperature sampling baseline. In contrast, mode-seeking methods, such as beam search and best-of-$N$, often generate sequences of higher probability but lower correctness. We interpret these findings through a variational objective that identifies power sampling as a soft alternative to mode-seeking. Finally, we demonstrate two applications of utilizing the probability-correctness relationship: improved inference-time scaling via power self-consistency and improved downstream performance through self-distillation.