Embrace the Diversity: Avoiding Mode Collapse with Polarized Curation in Generative Retraining

Retraining generative models on curated synthetic data, typically selected based on reward signals such as helpfulness and harmlessness, can amplify biases and trigger mode collapse—a failure mode in which the model collapses onto a narrow subset of outputs, losing diversity—especially when training data selection is guided by a monolithic function, such as maximizing alignment with a specific user preference. We introduce \emph{polarized curation}, a retraining framework in which data curation follows two reward functions, such as realism versus creativity, to reflect heterogeneous user preferences. We prove that polarized curation avoids collapse: the model distribution converges to a stable mixture over disjoint high-reward regions corresponding to the distinct preferences, retaining output variance and preserving diversity. Moreover, we reveal a game-theoretic structure underlying the limiting distribution, as the unique solution to a weighted Nash bargaining problem, where competing reward functions (preferences) negotiate a fair allocation over the model's output space. Experiments on synthetic mixtures and real-world image datasets demonstrate that polarized curation maintains diverse coverage across the two high-reward modes, stabilizes output variance across retraining rounds, and enables fine-grained control over generative behavior, the balance of competing objectives, via a single tunable parameter. Our results suggest that principled curation strategies such as polarized curation can play a central role in building generative models that reflect diverse and potentially conflicting user intents.