Failure Modes in LLM Agents for Safety-Critical Procedural Control: A Trace-Level Analysis from Nuclear Reactor Operation
Abstract
LLM agents are increasingly proposed for safety-critical procedural control, but their failure modes have not been systematically characterized. We present a trace-level empirical analysis of single-agent and multi-agent LLM architectures on the IAEA-standardized nuclear reactor trip-and-restart scenario---a reproducible 26-step procedure with 66 discrete control actions and irreversible physical consequences. Across 100 trials per architecture, the multi-agent Leader-Planner-Executor hierarchy raises aggregate completion from 66.12% to 78.46%, but this single number hides very different failure profiles. We define three correctness failure modes---Context Overwriting (CO), Verification Omission (VO), Rejection Deadlock (RD)---and one resource efficiency metric, Retrieval Redundancy (RR), with operational definitions drawn from annotated traces. Paired trace analysis shows that each multi-agent mechanism mitigates specific failure modes while introducing new ones absent in the single-agent baseline. Architecture selection for safety-critical deployment should therefore be guided by failure-mode-specific evidence rather than aggregate accuracy alone; we describe a methodology for generating such evidence.