Threshold-Guided Optimization for Visual Generative Models

Aligning large visual generative models with human feedback is often performed through pairwise preference optimization. While such approaches are conceptually simple, they fundamentally rely on annotated pairs, limiting scalability in settings where feedback is collected as independent scalar ratings. In this work, we revisit the KL-regularized alignment objective and show that the optimal policy implicitly compares each sample’s reward to an instance-specific baseline that is generally intractable. We propose a threshold-guided alignment framework that replaces this oracle baseline with a data-driven global threshold estimated from empirical score statistics. This formulation turns alignment into a binary decision task on unpaired data, enabling effective optimization directly from scalar feedback. We further incorporate a confidence weighting term to emphasize samples whose scores deviate strongly from the threshold, improving sample efficiency. Experiments across both diffusion and masked generative paradigms, spanning three test sets and five reward models, show that our method consistently improves preference alignment over previous methods. These results position our threshold-guided framework as a simple yet principled alternative for aligning visual generative models without paired comparisons.

AI systems that create images or videos are often improved by learning which outputs people prefer. Many current training methods need comparisons between two outputs, such as asking a person to choose the better image. These comparisons can be expensive and inconvenient to collect, while real feedback often comes as simple scores, such as star ratings or quality scores. This paper introduces Threshold-Guided Optimization, a way to train visual generation models directly from these individual scores. The method first looks at the score distribution and sets a data-driven cutoff. Outputs above the cutoff are treated as examples to encourage, while outputs below it are treated as examples to discourage. Scores far from the cutoff are given more weight, because they provide clearer feedback than borderline cases. Across image and video generation experiments, this approach improves alignment with preference-based quality measures without requiring explicit comparison pairs. The result is a simpler and more scalable way to use human or automatic feedback to improve visual generative models.