Keynote #2: Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Practitioners will often analyze a data set with the goal of applying any conclusions to a new population. For instance, one might hope that the top large language model (LLM) in a widely-used ranking system would outperform other ranked LLMs on future tasks. Typically the tasks that comprise the data used for ranking are not a perfect random sample from the population of future tasks --- but researchers might feel comfortable generalizing anyway so long as deviations from random sampling are small, and the corresponding impact on conclusions is small as well. Conversely, researchers might worry if a very small proportion of the data set was instrumental to the original conclusion (e.g. which LLM is top ranked). So we propose a method to assess the sensitivity of conclusions to the removal of a very small fraction of the data set. Manually checking all small data subsets is computationally infeasible, so we propose an approximation based on the classical influence function. Our method is automatically computable for common estimators, including those used in popular LLM rankings. We provide error bounds on approximation performance and a low-cost exact lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, does not disappear as data accrues, and is not decided by misspecification. Empirically we find that many data analysis conclusions are robust, but dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena.