ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

Scaling inference-time computation has enabled Large Language Models (LLMs) to achieve strong reasoning performance, but their inherently sequential decoding incurs substantial latency, motivating parallelization of the generation process. However, existing parallel reasoning approaches suffer from performance degradation compared to their sequential counterparts, and often rely on specialized inference engines. We introduce ThreadWeaver, a framework for adaptive parallel reasoning that matches the accuracy of comparably sized sequential reasoning models while significantly reducing inference latency via three key innovations: 1) a two-stage parallel trajectory generator that produces high-quality parallel chain-of-thought data for supervised fine-tuning; 2) a trie-based rollout design that enables parallel reasoning on any off-the-shelf autoregressive inference engine; and 3) a parallelization-aware reinforcement learning framework that trains the model to balance reasoning accuracy with effective parallelization. Across six challenging math reasoning benchmarks, ThreadWeaver trained on top of Qwen3-8B achieves performance on par with cutting-edge sequential reasoning models (79.9% on AIME24 and 71.9% on average) while delivering up to 1.53x speedup in token latency, establishing a new Pareto frontier between accuracy and efficiency.

Large language models can solve difficult problems by writing out many reasoning steps, but this can take a long time because they usually produce the steps one after another. This paper asks whether these systems can reason more like a team: when a problem has parts that can be considered separately, the model can split its work into several threads and later combine the results. We introduce ThreadWeaver, a way to teach a language model when to split its reasoning, when to keep working step by step, and when to bring separate pieces back together. The approach is designed to work with standard tools for running language models, rather than requiring special custom systems. On several challenging math tests, ThreadWeaver solves problems about as accurately as a strong model that reasons in the usual step-by-step way, while reaching answers faster. This suggests a practical path toward AI systems that can handle hard reasoning tasks with less waiting time, especially when extra computing power is available.