On the Difficulty of Learning a Meta-network for Training Data Selection

Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49\% over training without selection and 2.89\% over the strongest baseline.

Blindly using synthetic data for training can hurt model performance. A natural solution is to use a selection network to assign higher weights to useful examples and lower weights to less helpful ones. However, we find that this approach often performs worse than expected. In this work, we identify two key obstacles. First, the selection network suffers from a low gradient signal-to-noise ratio (GSNR). Our comprehensive analysis shows that the dynamics of training data weights directly lead to low GSNR, while also suggesting a simple remedy: increasing the batch size. Second, input features to the selection network do not correlate well with data effectiveness. We therefore design informative features that capture each example’s position in the training and validation distributions and its interaction with training. Intuitively, high-quality features ensure good solutions exist in the hypothesis space, or the set of all functions representable by the selection network; decent GSNR allows us to find such good solutions quickly.