Curated Synthetic Data Doesn’t Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

Recursive retraining of generative models poses a critical representation challenge: when synthetic outputs are curated based on a fixed reward signal, the model tends to collapse onto a narrow set of outputs that over-optimize that objective, causing diversity to vanish and failing to represent the full range of preferences. Prior work has suggested that such collapse is unavoidable without adding real data into the mix. In this paper, we revisit that conclusion from an alignment perspective and show that collapse can be mitigated through curation based on multiple reward functions. We formalize the dynamics of recursive training under heterogeneous preferences and prove that, under certain conditions, the model converges to a stable distribution that allocates probability mass across competing high-reward regions. The limiting distribution preserves diversity and provably satisfies a weighted Nash bargaining solution, offering a formal interpretation of value aggregation in synthetic retraining loops.

AI models are increasingly trained on (synthetic) data produced by earlier AI models. This can help when real data (human-written or human-labeled), but it also creates a problem. Previous work has shown that if each new round keeps only the outputs generated by the earlier model that satisfy only one preference, the model can collapse: its outputs become less varied over time, and in the limit, their variance can shrink to zero. We study a different way to choose synthetic data. Instead of using one fixed preference, we let the selection process switch between multiple preferences, such as different user tastes or different goals like quality, safety, and creativity. This small change has a large effect. Rather than concentrating on one narrow type of answer or image, the model can keep producing several kinds of desirable outputs. Our paper gives mathematical conditions under which this happens and tests the idea in synthetic examples, image generation, and text generation. The results suggest that collapse is not an unavoidable consequence of training on synthetic data. It depends on how that data is selected. More pluralistic selection can preserve diversity while still improving the model.