Reinforcement Learning with Promising Tokens for Large Language Models

Reinforcement learning (RL) has become a key tool for improving large language models (LLMs) on reasoning tasks. In practice, LLM rollouts are often generated with truncated decoding rules such as Top-k or Top-p sampling, whereas policy updates are still computed over the full vocabulary. This creates a support mismatch: data are collected from a compact set of plausible tokens, but the optimizer assigns learning signal through logits that were unavailable during rollout. In this work, we first show that successful reasoning tra- jectories are highly concentrated in the behavior model’s high-probability support, suggesting that rollout truncation can preserve most locally useful reasoning actions. We then propose Reinforcement Learning with Promising Tokens (RLPT), a support-aligned RL framework that constructs a state-dependent promising token set during rollout and reuses the same stored support during policy optimization. RLPT computes masked log-probabilities and policy ratios on the rollout support, aligning exploration and optimization on the same action space. Multi-seed experiments on mathematical reasoning benchmarks show consistent gains over GRPO, and ablations confirm that the improvement comes from support alignment rather than Top-k rollout alone.