When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

Foundation models are increasingly trained on synthetic data generated by prior model iterations rather than exclusively on real data. This self-consuming training paradigm can lead to model collapse, divergence, or bias amplification. Recent work (Ferbach et al., 2024) shows that incorporating human curation into the loop can steer a self-consuming model toward human-aligned behavior, but these analyses focus on a single, isolated model that solely consumes its own outputs. In practice, however, models often interact and train on input output pairs produced by other models. This paper studies self-consuming training in the multi-model regime. We first formalize a framework for interacting self-consuming models and characterize when the resulting dynamical system converges to a stable point. We then examine how human curation of one model affects its own alignment (self-influence) and how such effects propagate to other models (cross-influence). Unlike isolated settings where human curation always enhances model alignment, we show that cross-model interactions can dampen or even invert this effect, ultimately degrading long-term alignment.

AI models are increasingly trained not only on synthetic data generated by other models, but also on data curated by human. Previous studies mostly consider a single model, and human curation usually helps. People sometimes review or select the model-generated data before it's reused, with the hope that human curation will make future models better match human preferences. Previous studies mostly consider a single model learning from its own outputs, where human curation usually helps. This paper studies a more realistic setting where multiple models interact with each other and are trained on data produced by other models. In such systems, improving one model can indirectly affect other models, and those effects can later feed back into the first model. We show that human curation does not always enhance model alignment in this setting. Although curation may improve one model locally, interactions between models can weaken, distort, or even reverse its long-term alignment. Our results help explain when human curation is beneficial and when and how it may backfire in AI ecosystems where model-generated data is repeatedly reused. We support the analysis with experiments on simplified models, image generation and language models.