Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling

Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time. Particle Filtering (PF) has emerged as a strong ITS method for complex mathematical reasoning tasks, but it is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process. This causes PF to suffer from premature exploitation: it myopically commits to locally promising trajectories, prunes potentially correct hypotheses, and converges to suboptimal solutions. This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets. To address this, we analyze the problem and identify two root causes: a lack of diversity in the particle set due to overconfident resampling and consequent inability to assess the potential of a reasoning path. We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues. The first technique, Entropic Annealing (EA), directly mitigates particle impoverishment by monitoring search diversity via entropy; when diversity drops, it intervenes by dynamically annealing the resampling distribution to preserve exploration. The second, an enhancement called Look-ahead Modulation (LaM), adds a predictive guide to evaluate a state's potential based on its successors. By effectively balancing exploration and exploitation, ePF significantly outperforms strong baselines on challenging math benchmarks, achieving up to a 50% relative improvement in task reward.

Artificial intelligence models are increasingly used to solve complex tasks, like advanced mathematics, by exploring many possible solution paths at once. However, these models often suffer from tunnel vision. They can become too confident in early steps that look promising but eventually lead to wrong answers, causing them to ignore better alternatives. To fix this, we introduce Entropic Particle Filtering. Our method monitors the AI's search process and steps in when it narrows its options too quickly, forcing it to keep exploring diverse ideas. We also give the model a look-ahead ability to preview if a path is genuinely promising before fully committing to it. By preventing the AI from rushing to conclusions, our approach significantly improves its ability to solve highly difficult mathematical problems while using less computing power.