Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure

Latent or continuous chain-of-thought methods replace explicit textual rationales with a number of internal latent steps, but these intermediate computations are difficult to evaluate beyond correlation-based probes. In this paper, we view latent chain-of-thought as a manipulable causal process in representation space by modeling latent steps as variables in a structural causal model (SCM) and analyzing their effects through step-wise $\mathrm{do}$-interventions. We study two representative paradigms (i.e., Coconut and CODI) on both mathematical and general reasoning tasks to investigate three key questions: (1) which steps are causally necessary for correctness and when answers become decodable early; (2) how influence propagates across steps and how this structure compares to explicit CoT; and (3) whether intermediate trajectories retain competing answer modes and how output-level commitment differs from representational commitment across steps. We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing, and we identify a persistent gap between early output bias and late representational commitment. These results motivate mode-conditional and stability-aware analyses, together with corresponding training/decoding objectives, as more reliable tools for interpreting and improving latent reasoning systems. Code is available at https://github.com/J1mL1/causal-latent-cot.

Many large language models solve problems by writing out intermediate steps, much like a student showing their work. Newer systems try to do this reasoning internally, using hidden numerical representations instead of words. This can make reasoning faster, but it also makes the process much harder to inspect. Our work asks a simple question: when a large language model reasons in this hidden space, which hidden steps actually matter, and how do they affect the final answer? We study two recent hidden-reasoning methods, Coconut and CODI, by carefully changing one hidden step at a time and measuring whether the model’s answer or later internal states change. We find that hidden reasoning is not just a uniform stack of extra computation. Some steps have much more influence than others; information can jump across steps, and a model may appear to prefer an answer before its internal representation has fully settled.These findings give researchers tools to better diagnose hidden reasoning and suggest that future models should manage where information is stored, how it moves, and when a decision becomes stable.