When Sample Selection Bias Precipitates Model Collapse

The proliferation of recursive training on synthetic data can alleviate data scarcity but risks model collapse, where repeated training erodes distributional tails and homogenizes outputs. Data selection is widely viewed as a remedy, yet its reliability depends critically on the reference distribution used by the verifier. We show that in low-resource verification regimes, where each verifier observes only a small, fragmented, and biased slice of the target manifold, selection itself becomes biased. This situation naturally arises in low-resource data silos such as healthcare consortia or proprietary financial institutions, where raw data cannot be pooled and local references are inherently incomplete. As a result, selection preferentially retains samples aligned with the local manifold while pruning globally relevant tail modes, turning from a safeguard against collapse into a mechanism that precipitates it. We theoretically prove that such siloed selection accelerates collapse and induces power-law diversity decay. As an initial mitigation, we construct Wasserstein proxy references from multiple silos without sharing raw data. Empirical results confirm that local-reference selection fails on skewed distributions, whereas collaborative proxy references mitigate diversity degradation, suggesting that recursive synthetic-data pipelines require particular caution when real-data coverage is fragmented or scarce.

As AI-generated data becomes increasingly common, future AI systems may be trained on data produced by earlier AI systems. This creates a risk known as model collapse: over repeated training cycles, models may gradually lose rare or unusual examples and produce narrower, less diverse outputs. A common safeguard is to filter synthetic data before using it for training. Our paper shows that this safeguard can fail in low-resource settings. When a hospital, bank, or other data holder only has a small and biased view of the real world, its filter tends to keep synthetic samples that resemble its own limited data and discard samples that are still valid but less familiar. Over time, this biased filtering can accelerate collapse rather than prevent it, especially by removing rare or tail cases. We analyze this effect theoretically and verify it through experiments. As an initial mitigation, we propose using information from multiple data holders to form a shared reference without exchanging raw data, which helps preserve diversity during synthetic-data training.