Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework uses hierarchical Bayesian estimates, supports binary, ordinal, and continuous outcomes, and preserves full benchmark coverage without requiring a calibrated item bank. It runs live or retrospectively, and is more cautious for weak models where rare successes matter most. In a representative 200-item, 10-epoch evaluation, it removed 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, and therefore substantially reduced.