A Computational Framework for Evaluating Human-likeness in LLMs' Open-ended Human Behaviors

Large Language Models (LLMs) have found widespread application and research in scenarios such as role-playing and sociological simulations. Despite the growing use of LLM-based agents to simulate human activities, the extent to which their behaviors resemble human behavior remains underexplored. As diverse LLMs proliferate, the traditional Turing test is ineffective for scalable evaluation and prone to bias from human-crafted challenges, leading to unfair assessments. In this work, we propose a novel distribution-based framework that comprehensively evaluates human-likeness and believability of AI behaviors by leveraging large-scale open-ended human behavior data from web. For better evaluation, we design generic metrics to cover three principles: rationality, consistency, and diversity. Implemented across online shopping, open-topic Q&A, and urban mobility, our framework reveals that even the currently best LLM still exhibits a significant gap from real user behavior, underscoring the necessity of comprehensive research and evaluation of AI’s human-like capabilities.

AI systems are increasingly used to imitate human behavior, for example in games, online interactions, and studies of society. However, it remains unclear how closely their behavior matches that of real people. Current evaluations often rely on people judging a small set of examples, making them difficult to scale and sensitive to how the tests are designed. We propose a new evaluation approach that compares AI-generated behavior with large collections of real human behavior from the web. Our approach examines whether each AI-generated action resembles what a real person might do, whether its behavior remains coherent over time rather than shifting unpredictably, and whether a group of AI agents represents the varied behaviors found across real people. We apply it to online shopping, open-topic question answering, and urban mobility. Across these settings, even the strongest AI system we tested still differs substantially from real users. Our results show that careful evaluation is necessary before AI-generated behavior is used to represent or study human activity.