Pluralistic Leaderboards

Recent leaderboard-based evaluations of large language models aggregate user feedback by fitting a Bradley--Terry model to pairwise comparisons, producing a single global ranking based on a latent quality score. While appealing for its simplicity, this approach is incompatible with heterogeneous preferences: when LLMs are used across diverse tasks and use cases, users who favor fundamentally different model behaviors can be systematically misrepresented when collapsed into a single quality score. To address this issue, we study *pluralistic leaderboards* that aim to remain *stable* with respect to heterogeneous user populations. Drawing on ideas from social choice theory, we adapt the notion of *local stability*, which requires that no model outside the top-$k$ positions is collectively preferred to the top-$k$ set by more than $O(1/k)$ fraction of users. Building on techniques from the social choice literature, we design an alternative leaderboard mechanism that satisfies local stability while eliciting only $\widetilde{O}(k)$ pairwise comparisons per user, where $k$ is the size of the prefix for which stability is guaranteed. Using data from LMArena, we show that standard Bradley--Terry aggregation can violate local stability in practice, whereas our method provides substantially stronger stability guarantees.

Leaderboards are a central way of evaluating large language models, and many popular leaderboards are built from pairwise comparisons: users see two model responses and vote for the one they prefer. A common way to aggregate these votes is to fit a single quality score for each model and rank models by these scores. While simple and effective in many settings, this approach can be misleading when users have heterogeneous preferences: some may value safety and caution, while others may prefer creativity, conciseness, or detailed explanations. In such cases, there may be no single notion of model quality that faithfully represents what different users want. We study how to construct pluralistic leaderboards that better account for this heterogeneity. Drawing on ideas from social choice theory, we ask that the top-ranked models satisfy a representation guarantee: no sufficiently large group of users should collectively prefer an excluded model over all models shown near the top of the leaderboard. We design algorithms that achieve this kind of stability while eliciting only a limited number of pairwise comparisons from each user. Using simulations based on LMArena data, we show that standard Bradley–Terry aggregation can violate this stability requirement, whereas our methods produce rankings with substantially stronger representation guarantees. Our work suggests a path toward model evaluations that better reflect the diverse ways people use and judge AI systems.