Anytime-Valid Inference for Online Ranking of Large Language Models

Online evaluation of large language models increasingly relies on sequentially collected pairwise preferences, enabling human-aligned assessment and continuous data collection until closely performing models can be reliably distinguished. However, adaptive sampling and continuous monitoring invalidate classical fixed-sample inference, rendering existing ranking procedures largely heuristic. We propose SERPANT (Sequential E-value Ranking and Pruning via Adaptive Null Testing), a principled framework for online LLM ranking with anytime-valid guarantees. SERPANT formulates model comparison as a collection of pairwise hypothesis tests and constructs e-processes to ensure family-wise error rate control at any monitoring time. Anytime validity provides a theoretical justification for early stopping, enabling substantial cost savings from expensive human annotation. To improve efficiency, we introduce a novel tournament-based sampling strategy that adaptively selects comparisons based on past outcomes. The proposed framework is also extended to top-k model identification.

When large language models are compared online, evaluators often ask people which of two model outputs they prefer, and keep collecting comparisons until they are confident one model is better. This flexible process is useful, but it creates a statistical problem: traditional ranking methods assume the number of comparisons is fixed in advance, so their conclusions can become unreliable when researchers adaptively choose which models to compare or repeatedly check the results. We introduce SERPANT, a method for ranking language models that remains statistically reliable throughout an ongoing evaluation. SERPANT treats each comparison as a hypothesis testing and updates the evidence as more human preferences arrive, while controlling the chance of making false ranking claims. Because SERPANT can stop early when the evidence is strong enough, it can reduce the cost of expensive human annotation. It also chooses future comparisons in a tournament-like way, focusing effort on the models that are hardest and most important to distinguish. The result is a practical way to identify the best model and the top-performing models with confidence during continuous online evaluation.