Rethinking Thinking Tokens: LLMs as Improvement Operators

Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accuracy, but inflates context length, token/compute cost, and answer latency. We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier, e.g., better accuracy with lower context length and/or latency? Abstractly, we view the model as an improvement operator on its own "thoughts" with a continuum of possible strategies. We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace, producing an output that seeds the next round. Importantly, context length (hence compute cost) is controllable via degree of parallelism, and is no longer conflated with the total number of generated tokens. We report PDR instantiations of current models that give better accuracy than long CoT while incurring lower latency. Setting degree of parallelism to 1 yields an interesting subcase, Sequential Refinement (SR) (iteratively improve a single candidate answer) which provides performance superior to long CoT. Success of such model orchestrations raises the question whether further training could shift the Pareto frontier. To this end, we train an 8B thinking model with Reinforcement Learning (RL) to make it consistent with PDR as the inference method. On math tasks with verifiable answers, iterative pipelines surpass single-pass baselines at matched sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025).

Large language models often use very long step-by-step reasoning before writing an answer for very hard problems. Although this makes them more accurate, it also makes each answer more slower and expensive to generate. We study whether these models can be leveraged to benefit from careful reasoning without always writing a very long thought process. Our method, Parallel-Distill-Refine, asks the models to attempt several possible answers for a given problem at the same time, combine their useful parts into a single short summary note, and then improve the answer using that note. Repeating this process helps the models in iterating and revising their work while keeping the amount of generated thinking under control. On various reasoning benchmarks, these approaches outperform standard long thinking and reasoning while reducing the wait time to generate an answer. This suggests that future more capable models may become both more accurate and more efficient by learning better ways to organize their own reasoning.