Nonparametric LLM Evaluation from Preference Data

Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality with complex human responses such as ties. DMLRank comes with the following advantages: (i) It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows to incorporate black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.

Large language models are often compared using human preferences: two models answer the same prompt, and a person or automated judge decides which answer is better. These comparisons are then used to build leaderboards. However, current leaderboard methods can give misleading rankings when their assumptions are too simple, and they often do not clearly show valid uncertainty of rankings if machine learning models are used. This paper introduces a new framework for comparing language models from preference data. The framework supports several common ways of turning pairwise comparisons into model rankings, while also providing uncertainty estimates for the resulting scores. It can use flexible machine learning tools to learn from prompts and past comparisons, and it can also incorporate cheaper automated judges alongside human feedback. In addition, the paper shows how to choose future comparisons more efficiently when evaluation budgets are limited. Overall, the goal is to make language model leaderboards more reliable, more flexible, and less costly to build.