Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

Text-to-image models can create images from written prompts, but they still need training to better match human preferences. A common way to do this is to show the model two generated images and teach it to prefer the better one. However, existing preference datasets usually keep only the final images, while ignoring the random starting points that produced them. For rectified flow models, a modern type of image generator, this missing information matters because each image follows a simple path from its initial noise to the final picture. We propose Prior Noise-Aware Preference Optimization (PNAPO), a method that keeps both the preferred and less preferred images together with their original noise samples. This lets us better reconstruct how each image was generated, rather than relying on a separate and potentially inaccurate noise-adding process. PNAPO also adjusts how strongly the model learns from each preference pair, depending on how clear the preference is and how far training has progressed. Experiments on strong text-to-image models show that PNAPO improves alignment with human preferences while using much less training computation than previous methods. This makes preference training for image generation models more accurate, stable, and efficient.