Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a novel formulation showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over $10\times$ compared to MCMC-based sampling.

Large language models can often solve difficult reasoning problems, but improving their reliability typically requires reinforcement-learning post-training or costly inference-time sampling. This work shows that many of these gains can instead be recovered by changing how a base model samples its answers, without updating its parameters or using external verifiers. We propose a scalable power sampling method that encourages the model to follow reasoning paths that are more probable as complete trajectories, rather than simply choosing locally likely next tokens. During generation, our method estimates which candidate tokens are likely to lead to high-quality future continuations, yielding a practical approximation to the globally sharpened power distribution while avoiding the repeated refinement steps required by MCMC sampling. Across math, coding, and question-answering benchmarks, our method improves over standard decoding and other reward-free baselines, often matching or surpassing one-shot GRPO. It also reduces inference latency by over $10\times$ compared with MCMC-based power sampling. Overall, this work provides a training-free and verifier-free approach for improving LLM reasoning at inference time, offering a more efficient alternative to costly post-training and MCMC-based sampling.