Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models

Large Reasoning Models (LRMs) have recently achieved strong mathematical and code reasoning performance through Reinforcement Learning (RL) post-training. However, we show that modern reasoning post-training induces an unintended exploration collapse: temperature-based sampling no longer increases pass@$n$ accuracy. Empirically, the final-layer posterior of post-trained LRMs exhibit sharply reduced entropy, while the entropy of intermediate layers remains relatively high. Motivated by this entropy asymmetry, we propose Latent Exploration Decoding (LED), a depth-conditioned decoding strategy. LED aggregates intermediate posteriors via cumulative sum and selects depth configurations with maximal entropy as exploration candidates. Without additional training or parameters, LED consistently improves pass@1 and pass@16 accuracy by 0.61 and 1.03 percentage points across multiple reasoning benchmarks and models. Relevant code is included in the supplementary material and will made be fully public after this paper is accepted.

Large language models are becoming increasingly capable at solving complex reasoning tasks such as mathematics and coding. However, after being trained to reason better, these models often become overly confident in their first choice, which can reduce their ability to explore alternative solution paths. This is similar to a student who becomes too certain about an initial idea and stops considering other possibilities. In this work, we introduce a simple method that helps language models recover this lost flexibility during inference, without requiring any additional training. Instead of relying only on the model’s final prediction, our method also considers intermediate internal predictions made during the reasoning process, where the model often still preserves useful uncertainty and alternative options. Experiments on multiple reasoning and coding benchmarks show that this approach consistently improves performance across several state-of-the-art reasoning models. We further show that the same idea can also improve reinforcement learning training by encouraging more effective exploration during model optimization. Our results suggest that useful reasoning signals remain hidden inside language models even when their final outputs become overly confident, and that better use of these internal signals can improve both inference and training.