Goal-Drift Probes: Anticipating Multi-Turn LLM Agent Failure From Mid-Network Activations
Abstract
LLM agents executing multi-step tasks routinely fail by losing track of their objective; we ask whether such drift can be anticipated from the model's residual stream before the failing action is emitted. Instrumenting a Qwen2.5-7B-Instruct ReAct agent on ALFWorld, we train linear probes that decode the agent's task identity at 83.4% balanced accuracy and predict trajectory failure three steps in advance with AUC 0.989, far above behavioral-baseline, random-projection, and permutation-null floors. The signal survives a step-index residualization that linearly removes all step-index information from the activations (residualized AUC ≥ 0.939 at every layer), generalizes to held-out task types (mean AUC 0.99), to two independently-seeded held-out collections (AUC 0.88–0.91), to the entirely separate valid_seen ALFWorld split (~970 pooled trajectories, AUC 0.88–0.90), and replicates on Llama-3-8B-Instruct (AUC 0.97–0.98). A failure-mode decomposition finds pairwise direction cosines ≥ 0.79, indicating a single "about-to-fail" axis rather than mode-specific signals. The result is a concrete, deployable, and reproducibly-triggered model-internal monitor for agent goal drift.