From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents

Large Language Models (LLMs) are increasingly used as autonomous agents for reasoning and decision-making in interactive environments, yet the mechanisms guiding their step-by-step behavior remain unclear. This paper introduces a conformal interpretability framework for temporal tasks, which combines step-wise reward modeling with conformal prediction to label internal representations at each step as successful or failing. Using linear probes, we identify temporal concepts—latent directions in the model’s activation space associated with success, failure, or reasoning drift. Experiments in ScienceWorld and AlfWorld show these concepts are linearly separable and aligned with task outcomes. We also present preliminary evidence that steering models along successful directions can improve performance. Overall, this approach enables early failure detection and intervention, contributing to more interpretable and reliable LLM agents.