Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling

This paper investigates a pivotal yet debated component of reinforcement learning (RL) for training large language models (LLMs): controlling entropy (increasing or decreasing it) during RL fine-tuning. The existing literature presents a dichotomy: some studies posit that increasing entropy facilitates exploration, whereas others argue that decreasing entropy enhances performance. To reconcile these conflicting observations, we provide a theoretical framework showing that the effect of entropy is governed by \emph{Entropy Discrepancy}, the distributional divergence between positive and negative samples. Guided by this insight, we derive a principled dynamic scheduling method that adaptively modulates the entropy coefficient, effectively switching between entropy maximization and minimization as training evolves. Extensive experiments confirm the correlation between Entropy Discrepancy and the efficacy of entropy control. Furthermore, our adaptive method yields substantial improvements, boosting Pass@K by 6.7\% on AIME24 and 17.52\% on puzzle tasks compared to vanilla RL, while consistently outperforming recent state-of-the-art reasoning methods.

Modern AI systems like ChatGPT are increasingly trained on reasoning tasks — math problems, logic puzzles, coding — through a method called reinforcement learning, in which the AI tries answers, receives feedback on whether they were good, and gradually improves. A long-standing puzzle in this process is whether the AI should be encouraged to try more varied responses (which helps it discover new approaches) or to commit confidently to its best guess (which helps it lock in correct strategies). Different research teams have reported opposite conclusions, leaving practitioners without a clear rule to follow. This paper resolves that confusion. We show that the right choice depends on a single measurable quantity: roughly, how much the AI's uncertainty differs between problems it tends to solve correctly and problems it tends to get wrong. When that gap is large, encouraging variety pays off; when the gap is small, encouraging confidence pays off. The two seemingly conflicting bodies of prior work turn out to be describing two sides of the same coin. We then design a training procedure that monitors this quantity as the AI learns and automatically switches between the two strategies at the right moments — without any manual tuning. On math-competition-style benchmarks and logic-puzzle benchmarks, our method matches or outperforms carefully hand-tuned baselines as well as other recent specialized techniques, making reinforcement-learning training of language models more reliable and easier to deploy in practice.