Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuning

Fine-tuning pre-trained diffusion and flow models to optimize downstream utilities is central to real-world deployment. Existing entropy-regularized methods primarily maximize expected reward, providing no mechanism to shape tail behavior. However, tail control is often essential: the lower tail determines reliability by limiting low-reward failures, while the upper tail enables discovery by prioritizing rare, high-reward outcomes. In this work, we present Tail-aware Flow Fine-Tuning (TFFT), a principled and efficient distributional fine-tuning algorithm based on the Conditional Value-at-Risk (CVaR). We address two distinct tail-shaping goals: right-CVaR for seeking novel samples in the high-reward tail and left-CVaR for controlling worst-case samples in the low-reward tail. Unlike prior approaches that rely on non-linear optimization, we leverage the variational dual formulation of CVaR to decompose it into a decoupled two-stage procedure: a lightweight one-dimensional threshold optimization step, and a single entropy-regularized fine-tuning process via a specific pseudo-reward. This decomposition achieves CVaR fine-tuning efficiently with computational cost comparable to standard expected fine-tuning methods. We demonstrate the effectiveness of TFFT across illustrative experiments, high-dimensional text-to-image generation, and molecular design.

We study how to fine-tune generative models to optimize their tail performance, rather than their average performance. This is important because many real-world applications care about rare but consequential outcomes: scientific discovery often seeks rare, novel samples in the upper tail, while safe-critical settings require avoid rare but harmful failures in the lower tail. We propose an efficient method for directly optimizing these tail behaviors during fine-tuning. Our approach can either encourage a model to generate more high-reward outliers or reduce the occurrence of low-reward failures. We validate our method on tasks including novel molecule discovery and improving the worst-case quality of generated images.