When Preference Labels Fall Short: Aligning Diffusion Models from Real Data

Preference alignment aims to guide generative models by learning from comparisons between preferred and non-preferred samples. In practice, most existing approaches rely on preference pairs constructed from model-generated images. Such supervision is inherently relative and can be ambiguous when both samples exhibit artifacts or limited visual quality, making it difficult to infer what constitutes a truly desirable output. In this work, we investigate whether real data can serve as an alternative source of supervision for preference alignment. We adopt a data-centric perspective and study a curation strategy that treats real images as reference points and constructs preference signals by contrasting them with generated or perturbed samples, without requiring manually annotated preference pairs. Through empirical analysis, we show that real-data-based supervision provides effective guidance for aligning diffusion models and achieves performance comparable to existing preference-based methods. Our results suggest that real data offers a practical and complementary source of supervision for preference alignment and highlight directions of label-efficient alignment strategies. Code and models are available at https://cwyxx.github.io/RealAlign.

AI image generators are often improved by teaching them which images are “better,” typically by comparing pairs of generated images and indicating which one is preferred. However, this approach has an important limitation: when both images in a pair contain artifacts or have limited visual quality, the comparison may only reveal which imperfect image is relatively better, rather than what makes an image truly desirable. This makes it difficult for the model to infer how its generation process should be improved. In this work, we ask a simple question: instead of relying only on comparisons between imperfect generated images, can real images provide a clearer source of guidance? We treat real images as reference points and construct preference signals by contrasting them with generated images or deliberately degraded versions of real images. This allows the model to learn from real data without requiring humans to manually label which image is preferred in each pair. Our experiments show that this real-data-based approach can effectively guide diffusion models and achieve performance comparable to existing methods that depend on human-labeled preference pairs. The finding points to a practical and more scalable path for aligning generative models.