Embedding-perturbed Exploration Preference Optimization for Flow Models

Recent advancements have established Reinforcement Learning (RL) as a pivotal paradigm for aligning generative models with human intent. However, group-based optimization frameworks (e.g., GRPO) face a critical limitation: *the rapid decay of intra-group variance*. As the distinctiveness among samples within a group diminishes, the variance approaches zero. This eliminates the very learning signal required for optimization, rendering the process unstable and forcing the policy into *premature stagnation or reward hacking*. Existing strategies, such as varying the initial noise or increasing group sizes, often fail to address this fundamental issue, resulting in *training instability or diminishing returns*. To overcome these challenges, we propose **$E$mbedding-perturbed $E$xploration Preference Optimization ($E^2$PO)**, a novel framework that sustains optimization through embedding-level perturbation. Our method introduces structured, embedding-level perturbations within sample groups, guaranteeing a robust variance that preserves the discriminative signal throughout the training process. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art baselines, achieving a more faithful alignment with human preference.

Generative AI models learn human preferences by creating multiple examples and receiving rewards for the best ones. However, they often quickly start producing nearly identical outputs. When all options look the same, the AI cannot distinguish good from bad, causing the learning process to stall or rely on unhelpful shortcuts.To fix this, we introduce $E^2$PO, which applies small, structured "nudges" to the AI's underlying concepts during generation. This forces the model to continuously explore diverse possibilities rather than repeating itself. By maintaining these crucial differences between samples, training remains stable and active. Ultimately, this results in AI systems that more faithfully understand and generate what users actually want.