GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy

Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all tokens within the same response receive the identical reward. In this paper, we propose **Dynamic Entropy Weighting**, systematically define entropy-based weight ratios $\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$ and similar variants to redistribute rewards and get fine-grained rewards through two new algorithms: **Group Token Policy Optimization (GTPO)**, which assigns an entropy-weighted reward to each token and synthesizes token-specific advantage function to drive the model toward optimal path, and the analogous algorithm **Sequence-Level GRPO (GRPO-S)**, which admits a completely similar design at the sequence level. Unlike methods using entropy as mere regularization, GTPO and GRPO-S establish a new state-of-the-art on AIME and MATH 500, outperforming prior entropy-guided baselines and validating our weighting mechanism.

Large language models (LLMs) are often improved through reinforcement learning, but standard methods assign a single score to an entire response, making it hard to tell which specific words or reasoning steps were helpful. In this work, we address this by using the model’s own uncertainty, measured by a quantity called entropy, to automatically identify the most critical decision points and assign more precise, token-level feedback. We propose two algorithms: Group Token Policy Optimization (GTPO), which distributes rewards word by word based on uncertainty, and Sequence-Level GRPO (GRPO-S), a simpler variant that maintains stability over long reasoning chains. Our methods achieve state-of-the-art results on challenging math benchmarks including AIME and MATH 500. This work demonstrates that leveraging model uncertainty for fine-grained feedback is a promising step toward more reliable and sample-efficient reasoning in AI systems.