Semi-Supervised Learning with Noisy Proxy Covariates: Generalization Bounds and Distribution Regression

In many modern machine learning pipelines, abundant pretrained representations serve as noisy proxy covariates, while task-specific labels remain scarce. We study semi-supervised regression in this setting, and propose a simple two stage estimator that learns kernel eigenfeatures from all proxy covariates and fits a ridge predictor on labeled data. We derive finite sample bounds showing that fast labeled sample rates are recovered when proxy perturbation is controlled and unlabeled proxy covariates are sufficiently abundant. We also show that distribution regression is a direct special case, with analogous guarantees when the finite bag size is large enough. Experiments show consistent gains over supervised and semi-supervised baselines, especially in low label regimes.

In modern machine learning, it is often difficult and expensive to get enough perfectly labeled data, while large amounts of 'noisy' or imperfect data are easily accessible. This paper introduces a simple, two-step method that uses this abundant noisy data to learn useful underlying patterns, allowing us to make accurate predictions using only a small amount of labeled data. We mathematically prove that this method is highly efficient under the right conditions. Furthermore, we show that it can be directly applied to 'distribution regression'—a setting where predictions are made based on entire probability distributions or collections of samples. Our experiments confirm that this approach consistently outperforms existing techniques, especially when perfect data is very scarce.