A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

Understanding an agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a framework for evaluating goal-directedness that integrates behavioural evaluation with interpretability-based analyses of models' internal representations. As a case study, we examine an LLM agent navigating a 2D grid world toward a goal state. Behaviourally, we evaluate the agent against optimal policies across varying grid sizes, obstacle densities, and goal structures, finding that performance scales with task difficulty while remaining robust to difficulty-preserving transformations and multi-goal structures. We then use probing methods to decode internal representations of the environment and multi-step action plans. We find that the LLM agent non-linearly encodes a coarse spatial map, preserving approximate task-relevant cues about its position and the goal location; that its actions are broadly consistent with these internal representations; and that reasoning reorganises them, shifting from spatial cues toward immediate action selection. Our findings support the view that introspective examination is required beyond behavioural evaluations to characterise how agents represent and pursue their objectives.

As AI agents become more capable, we need better ways to understand not just what they do, but what drives their behaviour. An agent may fail because it has the wrong objective, because it misunderstands the situation, or because it cannot work out the right sequence of actions. These possibilities can look very similar if we only observe the agent from the outside. In this paper, we study a language-model agent navigating simple two-dimensional worlds. The agent must move through a grid to reach a goal, while sometimes encountering objects that resemble possible goals but should be ignored. We first evaluate the agent’s behaviour: how often it reaches the right goal, how performance changes as tasks become harder, and whether it remains consistent when the same problem is presented in different but equally difficult forms. We then look inside the model. By analysing its internal activity, we can recover rough "cognitive maps" of where the agent and goal are in the grid, as well as information about the actions the agent appears to be planning. These internal maps are imperfect, but they help explain the agent’s behaviour. In many cases where the agent appears to make a poor choice, its action is consistent with a mistaken internal picture of the world. Our results suggest that understanding AI agents requires both behavioural testing and internal analysis. Observing success and failure is not enough.