On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning. We release our code at https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/entropy.

Training large language models with rewards can improve their reasoning, but it can also make them too confident in a narrow set of answers, reducing their ability to explore different solutions. This paper studies why this loss of variety happens during reinforcement fine-tuning. We show that each training update can either increase or decrease the model’s output diversity, depending on whether the model is encouraged or discouraged to choose words that were already likely or unlikely. Based on this insight, we propose simple methods that prevent training from overly reducing diversity. Experiments on math reasoning tasks show that these methods help models explore more solution paths and achieve better performance.