Estimating the Empowerment of Language Model Agents

As language model (LM) agents become increasingly capable and adopted in real-world applications, there is a growing need for scalable evaluation frameworks beyond costly, manually designed benchmarks. We propose information-theoretic evaluation based on empowerment, an information-theoretic measure of an agent’s influence on future states through its actions. To handle the unique challenges of text-based environments, we introduce EELMA (Estimating Empowerment of Language Model Agents), an algorithm for approximating effective empowerment from multi-turn text interactions. We demonstrate EELMA on textual games and realistic web and tool-use environments, showing that empowerment strongly correlates with average task performance. We further analyze how empowerment varies across models, environment complexity, and agent configurations, and show that high-empowerment states and actions often mark pivotal moments for general capabilities. These results establish empowerment as a goal-agnostic metric that complements task-success measures for LM-agent evaluation. Code available: https://github.com/Jinyeop3110/EELMA

AI agents that browse websites, write code, or manage files are increasingly part of daily life, but measuring their true capabilities requires designing hundreds of specific test scenarios by hand, a process that is costly and may miss important behaviors. We propose a new measure called empowerment, which captures how many different futures an agent can navigate toward from any starting point. Think of it as assessing athletic versatility rather than scoring performance on a single sport: a truly capable agent can steer toward many outcomes, while a limited one gets stuck repeating the same actions. We developed a method that estimates empowerment from recordings of agent behavior alone, requiring no predefined tasks or goals. Empowerment reliably predicts how well agents perform on standard tasks across web browsing and customer service settings. It also automatically highlights moments when an agent gains unexpected access or influence, without needing to be told what to look for.