Inducing post-hoc chain-of-thought reasoning in LLMs on multiple-choice question answering tasks

Chain-of-thought (CoT) monitoring is a promising approach to ensuring the safety of large language models (LLMs). However, for monitoring to be effective, the CoT must be faithful: it must reflect the model's underlying reasoning process. If an LLM generates a CoT in order to justify a predetermined answer post-hoc, said CoT can be unfaithful. In this work, we provide evidence that LLMs from three different model families engage in post-hoc reasoning on multiple-choice question answering datasets. Specifically, we find that models record their belief in the correctness of each answer choice via a correctness direction located at the delimiter after the choice and the final few tokens of the choice; steering with this direction causes the model to confabulate a CoT that supports the steered-to answer. However, we find that post-hoc reasoning is less prevalent for questions involving step-by-step mathematical reasoning. Our work provides a preliminary mechanistic account of how pre-computed answers can drive unfaithfulness in LLMs.