Training Continuous Chain of Thought Models: A Tale of Two Regimes

Continuous Chain-of-Thought (ContinuousCoT) replaces verbose reasoning traces with dense latent representations. Training methods typically fall into two regimes: Indirect Supervision, which autoregressively generates latents during training, and Direct Supervision, which leverages CoT traces directly for fully parallel training. We introduce C-MTP, a direct supervision approach that predicts the tokens corresponding to each intermediate latent and aggregates their embeddings to feed into the next step, naturally extending next-token prediction without ad-hoc latent modeling. We show that our approach significantly outperforms existing direct supervision methods across diverse evaluation settings. We also conduct a systematic empirical study of both regimes across three CoT trace difficulty levels: Structured, Semi-Natural, and Realistic. On compact, structured traces, direct supervision remains preferable for its efficiency and generalization. On verbose traces, indirect supervision dominates; its recurrent objective better compresses long reasoning into a fixed latent budget, whereas direct methods struggle with train-test mismatch due to teacher forcing. On long realistic traces generated by LLMs, both regimes significantly underperform standard CoT fine-tuning, revealing fundamental limitations in current ContinuousCoT training.