Evaluating Language Models in Realistic Conversational Contexts

As Large Language Models (LLMs) are increasingly deployed to serve open-ended, multi-turn interactions, evaluating conversational quality at human scale has become a central challenge. Existing evaluation frameworks built for summarization, translation, or short-form QA tasks fall short of adequately measuring the consistency of human-scale dialogue, especially when derivation and validation of these metrics themselves often rely on synthetic rather than human sources. We fill the gap by introducing \textbf{UPHELD} (Utility & Planning Human-Scale Evaluated Long Dialogues), a large, reference-full benchmark for evaluating human-scale conversational ability beyond factual correctness. UPHELD consists of hundreds of complete human-to-human dialogues authored by \textbf{professional script writers}, with realistic turn densities and \textbf{36,000+ per-turn human annotations} across \textbf{30,000+ expert-generated dialogue turns}. Using UPHELD, we systematically evaluate classical automatic metrics and reference-free LLM-as-a-judge approaches, and find them unreliable when correlated with expert human judgment. Building off this analysis, we use UPHELD to develop a \textbf{Mixture-of-Judges} framework that combines multiple evaluative signals and improves correlation with human assessments by approximately \textbf{30\%}. Overall, UPHELD provides a robust, human-grounded foundation for evaluating human-scale conversational intelligence that fills a crucial gap in the pre-existing LLM dataset landscape.

Current datasets used to train and evaluate LLMs for human-like conversation are often inadequate, as they were not specifically designed for this purpose. To address this, we worked with professional writers to create a dataset of everyday conversations where two people collaborate to accomplish a real-world goal. After collecting and verifying these conversations, we show how models can use this data to improve and how automated evaluation methods can reliably mimic human judgment. Together, these contributions give researchers and practitioners a high-quality dataset they can use to build better conversational models or make their evaluations more robust.