Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design

Reinforcement learning has been widely applied to diffusion and flow models for visual tasks such as text-to-image generation. However, these tasks remain challenging because diffusion models have intractable likelihoods, which creates a barrier for directly applying popular policy-gradient type methods. Existing approaches primarily focus on crafting new objectives built on already heavily engineered LLM objectives, using ad hoc estimators for likelihood, without a thorough investigation into how such estimation affects overall algorithmic performance. In this work, we provide a systematic analysis of the RL design space by disentangling three factors: i) policy-gradient objectives, ii) likelihood estimators, and iii) rollout sampling schemes. We show that adopting an evidence lower bound (ELBO) based model likelihood estimator, computed only from the final generated sample, is the dominant factor enabling effective, efficient, and stable RL optimization, outweighing the impact of the specific policy-gradient loss functional. We validate our findings across multiple reward benchmarks using SD 3.5 Medium, and observe consistent trends across all tasks. Our method improves the GenEval score from $0.24$ to $0.95$ in $90$ GPU hours, which is $4.6\times$ more efficient than FlowGRPO and $2\times$ more efficient than the SOTA method DiffusionNFT without reward hacking.

Text-to-image AI systems can create impressive images from written prompts, but making them better is still difficult. A common method is to generate many images, score them, and then train the system to produce higher-scoring images in the future. However, this process can be unstable because it is hard to know how strongly the system supported each image it generated. In this work, we study what matters most when improving these image generators with scores. We compare different ways of using the scores, different ways of estimating the system’s confidence in its images, and different ways of generating images during training. We find that the most important part is getting a reliable estimate of the system’s confidence, rather than choosing a more complicated training rule. Based on this finding, we develop a more stable and efficient way to improve text-to-image systems. Our method improves image quality while using less computing power than prior approaches. This suggests that future progress may come from better understanding and measuring how these systems generate images, not only from designing new training rules.