Chain-of-Thought Reasoning In The Wild Is Not Always Faithful

Recent studies indicate that when faced with explicit biases in prompts, models often omit mentioning these biases in their Chain-of-Thought (CoT) output, revealing that verbalized reasoning can give an incorrect picture of how models arrive at conclusions (unfaithfulness). In this work, we show that unfaithful CoT also occurs on naturally worded, non-adversarial prompts without adding artificial biases or editing model outputs. We find that when separately presented with the questions "Is X bigger than Y?" and "Is Y bigger than X?", models sometimes produce superficially coherent arguments to justify systematically answering Yes to both or No to both, despite the contradiction. We present preliminary evidence that this is due to models' implicit biases towards Yes or No, labeling this Implicit Post-Hoc Rationalization. Our results reveal rates up to 13% for production models, and while frontier models are more faithful, none are entirely so, including thinking models like DeepSeek R1 (0.37%) and Sonnet 3.7 with thinking (0.04%). We also investigate Unfaithful Illogical Shortcuts, where models use subtly illogical reasoning to make speculative answers to hard math problems seem rigorously proven. Our findings indicate that while CoT can be useful for assessing outputs, it is not a complete account of the internal process that produced the model's answer and should be used with caution in agentic or safety-critical settings.

Large language models are often asked to explain their reasoning step-by-step before giving an answer, a process called Chain-of-Thought (CoT). Many hope CoT makes AI systems easier to oversee, since a human can check whether the answer follows from the explanation. We investigated whether the explanations these models give actually match how they arrived at their answers. We tested 15 popular AI models from OpenAI, Google, Anthropic, Meta, Qwen, and DeepSeek, on simple comparative questions and on difficult mathematics problems, and found two kinds of "unfaithful" explanations that arise on ordinary prompts, without any prompt manipulation. First, when asked logically opposite versions of a comparison question, like "Is X bigger than Y?" and "Is Y bigger than X?", some models give the same yes-or-no answer to both, inventing different-looking reasons each time to support their preferred answer. Second, when solving difficult math problems, models can land on the correct final answer while making clearly invalid logical jumps in their explanation, never acknowledging the jump. These failures suggest AI explanations should be read as helpful but incomplete: they can flag problems but should not be trusted as the full account of how a model decided.