Select to Think: Unlocking SLM Potential with Local Sufficiency

Small language models (SLMs) offer efficient deployment, yet they often lag behind their larger counterparts (LLMs) in reasoning. Existing remedies either invoke an LLM at points of reasoning divergence, incurring substantial latency and cost, or rely on standard distillation, which is limited by the SLM's capacity to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token often resides within the SLM's top-$K$ next-token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose Select to Think (S2T), which reframes the LLM’s role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-Local, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, a 1.5B SLM's top-8 candidates contain the 32B LLM's choice with a 95\% hit rate, and S2T-Local improves the 1.5B SLM's Math Avg. over greedy decoding by 24.1\% relative gain, matching the efficacy of 8-path self-consistency with single-trajectory efficiency. Code is available at https://github.com/YeRona/Select-to-Think.

Small language models (SLMs) are attractive for their efficiency, yet they often fall short of the reasoning power of larger language models (LLMs). Existing solutions either ask an LLM for help during inference, making reasoning slower and more expensive, or distill the LLM into the SLM, which can be too demanding due to the capacity gap. Our key finding is local sufficiency: at difficult decision points, LLM’s preferred next step is often already present among the SLM’s top few candidate tokens, even when SLM does not rank it first. In our experiments, the 32B model’s preferred token appears in the 1.5B model’s top-8 candidates 95% of the time. This means that SLM may not need LLM to generate new reasoning tokens; instead, it mainly needs help choosing among options it already considered. Select to Think therefore changes the LLM’s role from generator to selector over the SLM’s own candidates. We then train this selection ability into the SLM, producing S2T-Local, which re-ranks its own candidates without LLM calls at test time. This substantially improves single-trajectory reasoning by an average of 24% relative gains on 1.5B SLM.