Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models

Current LLM evaluations often conflate benchmark performance with intrinsic model capability. This is misleading, as observed outcomes arise from the entire evaluation system, including datasets, prompting methods, decoding parameters, and the software–hardware stack, rather than the model alone. When this system is under-specified, attribution becomes unreliable; in practice, evaluation choices alone can induce accuracy swings of up to 70%. This challenge is compounded by the open-ended nature of LLM evaluation, where questions span languages, domains, and usage styles, forming variable and implicitly shifting datasets. Consequently, strong performance on static benchmarks may reflect surface alignment or dataset-induced effects rather than robust capability. Prior studies often focus on individual components or manually-curated small-scale dataset variants, overlooking interactions and dataset-related confounding. To address these limitations, we propose LLM evaluatology, a principled framework that grounds LLM evaluation in a causally motivated system design. It combines structured causal modeling as an intervention-oriented lens with factorial decomposition under design of experiments, quantifying main and interaction effects while using instance-level interventions to probe dataset-induced effects. By jointly modeling evaluation components and structured question variations, LLM evaluatology enables more interpretable, reproducible, and carefully attributed assessment of model capability. Our framework is publicly available at GitHub.

When people test large AI language models, a single benchmark score is often treated as a measure of how "smart" the model is. Our research shows this can be misleading. The same model can score anywhere from near zero to about 90% correct, depending on choices that have little to do with the model itself—such as how questions are formatted, whether the model is asked to "think step by step," or how long its answers are allowed to be. These evaluation choices can substantially shift reported accuracy, and the ranking of which model is "best" can even reverse. A model that answers an original question correctly may fail when the question is slightly reworded or perturbed, suggesting memorization rather than robust reasoning. We developed a framework called LLM evaluatology to address this. Instead of testing a model once under a default setting, we treat evaluation as a full system: we systematically vary key evaluation choices, automatically generate new question variants (including transformed and newly synthesized items), and use statistical analysis to identify which factors most strongly drive performance. This work matters because it offers a more reliable way to compare AI systems and to understand what benchmark scores actually reflect. It encourages the field to move beyond one-number leaderboards toward evaluation that is more transparent, reproducible, and faithful to real-world use.