ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

Large Reasoning Models (LRMs) have achieved remarkable progress thanks to Reinforcement Learning with Verifiable Rewards (RLVR) on Chain-of-Thoughts (CoTs). However, since long CoTs naturally contain trial and errors and mainstream RLVR approaches choose outcome-correct CoT trajectories for memorization, the redundant explorations in long CoTs are inevitably reinforced, which results in the over-thinking issues of LRMs. Previous attempts to resolve this issue mainly give more advantage to shorter trajectories, yet their learning signals are still outcome-based and cannot reduce the memorization of redundant explorations in long CoTs. Therefore, we propose ThoughtFold, a framework that leverages fine-grained preference learning to mitigate redundant explorations for efficient reasoning. ThoughtFold employs an introspective strategy to identify redundancy within each correct trajectory, which yields a spectrum of candidate sub-trajectories. Leveraging this spectrum, we introduce a masked preference optimization objective that explicitly penalizes redundant explorations and encourages the model to directly bridge essential reasoning segments, effectively folding its reasoning chains into a more concise path. Extensive experiments show that ThoughtFold significantly enhances efficiency. It reduces the token usage of DeepSeek-R1-Distill-Qwen-7B by approximately 56\% while maintaining state-of-the-art accuracy.

Large Reasoning Models can solve hard problems by writing long chains of intermediate reasoning. But these chains often include redundant explorations, such as repeated checks, hesitation, or off-target attempts. Because current training methods mainly reward correct final answers, they can also reinforce these unnecessary steps. We propose ThoughtFold, a framework for efficient reasoning. It uses an introspective strategy to identify redundant explorations within correct reasoning trajectories and turns them into fine-grained preference signals. These signals penalize unnecessary reasoning steps and encourage the model to directly connect essential reasoning segments, effectively folding long reasoning chains into more concise paths. Our analysis shows that the improvement is more than shorter output. ThoughtFold makes the model’s reasoning more concentrated, with fewer loops and detours. This suggests that reasoning models can learn to think more efficiently while preserving the reliable problem-solving behavior that long reasoning chains are meant to provide.