iGRPO: Fast Online RL for Flow Matching Model with Instant Reward

Conventional practice assumes that online reinforcement learning for flow-matching models requires sampling full denoising trajectories to compute rewards. This assumption underlies methods such as Group Relative Policy Optimization (GRPO), where the policy must traverse the entire reverse process before receiving a delayed, trajectory-level reward. We observe, however, that while such terminal rewards provide feedback, they are neither necessary nor optimal for effective learning. In this work, we introduce iGRPO (Instant-reward GRPO), which replaces GRPO's full-trajectory rollouts with a single-step mapping that assigns rewards instantly at each denoising step. Because the flow matching model behaves differently across timesteps, our step-local instant rewards which are inherently time-dependent, overcome prior approaches that rely on a single, time-independent terminal reward. By evaluating each action locally rather than relying on a final terminal score, iGRPO eliminates the need for multi-step SDE rollouts and offers more precise credit assignment. Across standard benchmarks, iGRPO converges 10.2× faster than FlowGRPO while achieving higher final alignment quality. We hope this work motivates more efficient and scalable online RL methods for flow-matching generative models.

Modern image-generation systems can create impressive pictures from text, but they still often struggle with details such as the correct number of objects, spatial relationships, attributes, and readable text. One way to improve these systems is to let them learn from feedback, similar to how a person improves after being told which outputs are better. However, existing methods usually require the model to generate a full image before receiving feedback, making training slow and expensive. In this paper, we introduce iGRPO, a faster way to train image-generation models with feedback. Instead of waiting until the entire image-generation process finishes, iGRPO gives the model feedback immediately at each step by quickly estimating what the current step would produce. This allows the model to understand which decisions are helpful much earlier, while avoiding many unnecessary computations. Across several image-generation benchmarks, iGRPO improves text-image alignment and compositional accuracy while using much less training time than prior methods. This makes it easier and cheaper to adapt powerful image-generation models for creative, educational, and practical applications. At the same time, stronger and cheaper image generators also require careful safety measures to reduce misuse such as misleading or harmful content.