Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling

Many reinforcement learning (RL) problems admit multiple terminal solutions of comparable quality, where the goal is not to identify a single optimum but to represent a diverse set of high-quality outcomes. Nevertheless, policies trained by standard expected-return maximization routinely collapse onto a small subset of outcomes, a phenomenon commonly attributed to insufficient exploration or weak regularization. We show that this explanation is incomplete: outcome-level mode collapse is a structural consequence of the expected-return objective itself. Under idealized learning dynamics, the log-probability ratio between any two outcomes evolves linearly in their reward difference, implying exponential ratio divergence and inevitable collapse—independent of the exploration strategy, entropy regularization, or optimization algorithm. We identify the source of this pathology as the probability multiplier inside the expectation and propose a minimal correction: inverse probability scaling, which removes outcome-frequency amplification from the learning signal, fundamentally changes the learning dynamics, and provably yields reward-proportional terminal distributions, preventing collapse in multimodal settings. We instantiate this principle in Group Relative Policy Optimization (GRPO) as a drop-in modification, IPS-GRPO, requiring no auxiliary models or architectural changes. Across different reasoning and molecular generation tasks, IPS-GRPO consistently reduces outcome-level mode collapse while matching or exceeding baseline performance, suggesting that correcting the objective rather than adding exploration heuristics is key to reliable multimodal policy optimization.

Many reinforcement learning systems are trained to find actions that receive high rewards. In many real-world problems, however, there is not just one good answer. For example, a language model may have many valid ways to solve a reasoning problem, and a molecule-generation model may be able to produce many different useful molecules. A good system should therefore find many high-quality solutions, not collapse to only one or two. This paper studies why such collapse happens. We show that the usual reinforcement learning training objective naturally gives more learning signal to outcomes that the model already produces more often. This creates a feedback loop: common outcomes become even more common, while other good outcomes are gradually ignored. This can happen even when the ignored outcomes are just as good. We propose a simple correction called Inverse Probability Scaling. The idea is to reduce the extra advantage given to outcomes simply because they are already frequent. This encourages the model to spread probability across multiple good outcomes in proportion to their quality. We implement this idea in a method called IPS-GRPO, which can be added to existing training pipelines with minimal changes. Across controlled grid tasks, reasoning benchmarks, and molecule-generation experiments, IPS-GRPO finds a wider range of high-quality outcomes than standard reinforcement learning baselines while maintaining strong performance. These results suggest that improving the training objective itself can be an effective way to preserve diversity in reinforcement learning systems.