Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action’s ``pure" effect, and (ii) it identifies turning points—steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend—and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation.

Text-to-image models create images through many small denoising steps, but current reinforcement learning methods usually judge all of these steps only by the final image quality. This makes it hard to tell which individual steps actually helped and which ones hurt. It also misses an important effect: some early steps may not look useful immediately, but can strongly influence later progress. To address this, we developed TurningPoint-GRPO, a new training method for flow matching image generators. Instead of giving every step the same final reward, our method assigns each step an incremental reward based on how much it improves the image at that moment. We also identify “turning points”, steps where the direction of progress changes and later improvements begin to align with the overall trajectory. These turning points receive extra long-term credit so the model can learn delayed effects, not just immediate ones. Importantly, turning points are detected automatically from reward trends, without extra tuning. Across experiments, this leads to more effective use of reward signals and more consistent improvements in image generation quality. Our work matters because it makes reinforcement learning for text-to-image models more precise, interpretable, and effective.