Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens

While explicit Chain-of-Thought (CoT) equips Large Language Models (LLMs) with strong reasoning capabilities, it constrains the model's thoughts to a discrete vocabulary space. Recently, reasoning in continuous latent space has emerged as a promising alternative, but current paradigms suffer from feature collapse and instability due to distribution mismatch when recurrently reusing hidden states, or alignment issues when relying on assistant models. To address this, we propose $\textbf{Latent Thoughts Tuning (\textit{LT-Tuning})}$, a post-training framework that redefines how latent thoughts are constructed and deployed. Instead of relying solely on raw hidden states, our method introduces a $\textbf{Context-Prediction-Fusion}$ mechanism that jointly leverages contextual hidden states and predictive semantic guidance from the vocabulary embedding space. Combined with a progressive three-stage curriculum learning pipeline, $\textit{LT-Tuning}$ also enables dynamic switching between latent and explicit thinking modes. Experiments demonstrate that our method outperforms existing latent reasoning baselines, effectively mitigating feature collapse and achieving robust reasoning accuracy.

AI language models like the ones powering chatbots solve hard problems by writing out every reasoning step in plain text, much like a student showing their work. This is reliable but slow and wordy, and it forces the AI's thinking through the bottleneck of everyday language. We wanted to teach AI to think more like a person does internally, without verbalizing every step. Previous attempts at this "silent thinking" suffered a strange failure: the AI's internal thoughts tended to collapse into indistinguishable blobs, especially in larger models, which made the AI worse rather than better. Our method, Latent Thoughts Tuning, fixes this by building each silent thought from two ingredients: a summary of what the AI has been processing so far, and a hint of what it would most likely say out loud next. We also teach the AI to decide on its own when to think silently versus when to spell things out, based on how confident it feels. On standard math reasoning tests, this approach delivers more accurate answers while producing shorter responses. It is a step toward AI that can mix silent and spoken thought, the way humans naturally do.