Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL

Reinforcement-learning-based post-training for large language models vastly improves capabilities on verifiable tasks however it suffers from training instability and diversity collapse. As a solution, many new advantage functions have emerged. Each method simultaneously changes which problems receive gradient, the balance between positive and negative updates, and the overall gradient scale making their comparison difficult. We propose a unifying framework that decomposes each function by their induced positive and negative gradient mass. Analyzing policy performance and the geometry of model weights reveals: symmetric weights enhance diversity whereas asymmetric weights learn faster. Combining the advantages of both, we build a new adaptive advantage: FADE which is fast, diverse and accurate at various model scales (7B, 32B).