Towards Agentic Failure Analysis: A Minimal Trace-Level Diagnostic for Chain-of-Thought Reasoning

Single-turn chain-of-thought (CoT) prompting is often evaluated by final-answer accuracy. Under injected bias, however, accuracy can collapse into two distinct cases: a model may give the same final answer while either referencing the injected content in its reasoning trace or not mentioning it at all. We study this as a measurement blind spot and introduce a minimal trace-level diagnostic for separating these cases. For each biased trial, we measure two axes: susceptibility (whether the bias breaks a previously correct answer) and acknowledgment (whether the trace contains a rubric-defined surface reference to the injected content). Across thousands of biased GSM8K trials, GPT-4o and Claude Sonnet 4 have similar susceptibility rates (1.3% vs. 1.2%) but substantially different acknowledgment rates (13.0% vs. 75.0%) under the same rubric. We do not evaluate multi-step agent systems. Instead, we use single-turn CoT under bias as a controlled local-step setting for measuring trace-level behavior, with the goal of informing future failure analysis in longer agentic trajectories.