Discovering Differences in Strategic Behavior between Humans and LLMs

As Large Language Models (LLMs) are increasingly deployed in social and strategic scenarios, it becomes critical to understand where and why their behavior diverges from that of humans. While behavioral game theory (BGT) provides a framework for analyzing behavior, existing models do not fully capture the idiosyncratic behavior of humans or black-box, non-human agents like LLMs. We employ AlphaEvolve, a cutting-edge program discovery tool, to directly discover interpretable models of human and LLM behavior from data, thereby enabling open-ended discovery of structural factors driving human and LLM behavior. Our analysis on iterated rock-paper-scissors reveals that frontier LLMs can be capable of deeper strategic behavior than humans. These results provide a foundation for understanding structural differences driving differences in human and LLM behavior in strategic interactions.

LLMs are increasingly deployed within interactive situations with humans, or as stand-ins for humans in social science research, but we do not fully understand how their decision-making differs from that of humans. To study this, we had humans and several frontier LLMs play rock-paper-scissors against opponents with fixed, exploitable strategies, such that consistently winning would require detecting and adapting to patterns in the opponent's behavior. We then used AlphaEvolve to automatically generate readable programs that model how an agent makes decisions, to compare the underlying strategies of humans and AI systems. Frontier LLMs outperformed humans by identifying opponent patterns faster. The discovered programs reveal that this difference is likely because the frontier LLMs track more complex and complete histories of opponent gameplay than humans do. These findings demonstrate that frontier AI systems are not reliable stand-ins for human behavior in strategic settings. Further, their superior ability to model opponents could carry risks in high-stakes interactions.