Benchmarking World-Model Learning with Environment-Level Queries

World models are central to building AI agents capable of flexible reasoning and planning. Yet current evaluations (i) test only properties measurable from observed interactions, such as next-frame prediction or task return, and (ii) do not test whether a learned model supports diverse queries about the environment. In contrast, humans build general-purpose models that can answer many different questions about an environment—including questions that require understanding global structure and counterfactual consequences. We propose WorldTest: a protocol for evaluating whether agents learn models that support multiple environment-level queries—questions whose answers depend on properties of the full environment, not just observed trajectories. Individually, these queries can target properties (e.g., reachability or the effects of interventions) that no single rollout distribution determines. Collectively, they assess model generality across query types. We instantiate WorldTest as AutumnBench, a benchmark of 43 interactive grid-world environments and 129 tasks across three query families for both humans and learning agents. Experiments with 517 human participants and five frontier models on AutumnBench show that humans substantially outperform these models, a gap we attribute to differences in exploration and belief updating.

Consider someone who cooks regularly in their own kitchen. Over time, they build an internal model of the workspace, such as where tools live and how appliances behave, that supports various everyday capabilities. For example, it enables the person to: (1) predict how long the hidden contents of a covered pot will take to finish cooking, based on steam intensity and elapsed time, (2) plan a sequence of actions to complete a set of recipes, and (3) recognize and adapt to changes in the kitchen (e.g., when knives are placed in a different drawer). While existing benchmarks test some of these capabilities in AI agents, there is no single framework to capture the core capabilities of world model learning. For instance, an agent can accurately generate a video of a cook at work, but will still be unable to translate that into a recipe for the dish. Our main motivation is to evaluate these capabilities in a way that is generalizable to all agents, including humans. We therefore build WorldTest, a framework for evaluating world-model learning with environment-level queries: questions about the environment as a whole. WorldTest does this in two phases. The first phase allows the agent to explore the environment freely, with neither rewards nor goals. Then the second phase asks the agent to answer a question by posing it as a derived challenge environment: a new environment whose explicit objective is solvable only by an agent that has a correct world model. Our benchmark, AutumnBench, applies WorldTest to 43 interactive grid worlds and three query types: prediction, planning, and change detection. We ran the benchmark with 517 human participants and five frontier reasoning models. Humans beat frontier reasoning models by a wide margin on AutumnBench, revealing gaps in belief updating and figuring out what caused some outcome during exploration.