Coupled Training with Privileged Information and Unlabeled Data

In many prediction problems, we have extra information during training (for example, measurements that are expensive or slow to collect) that will not be available when the model is deployed. A common strategy is to first train a model that uses all training information, then use its predictions on unlabeled examples to train a second model that only uses the inputs available at test time. However, when the extra training-only information is weak or noisy, this two-step approach can mislead the deployable model and even hurt accuracy. We propose a joint training method that learns the two models together, so the deployable model can benefit from the extra information only when it actually helps, instead of inheriting its mistakes. We provide guarantees that describe when joint training improves prediction accuracy and analyze a simple alternating training algorithm for large, high-dimensional models. Experiments on synthetic data and medical prediction tasks show that our approach avoids these failures and consistently outperforms standard two-step baselines.

Many prediction systems are trained with information that will not be available when the system is actually used. For example, a medical model may be trained using expensive lab tests or expert assessments, even though the final tool must make predictions from cheaper routine measurements. A common approach is to first train a model using all available information, then use its guesses to train a second model that only uses the information available in practice. However, this can go wrong when the extra training-only information is noisy or only weakly helpful: the final model may learn from bad guesses and perform worse. We propose a new training method that learns these two models together instead of one after the other. This allows the final model to benefit from the extra information when it is useful, while limiting its influence when it appears unreliable. We provide mathematical results explaining when this joint approach should improve prediction accuracy, and we develop an efficient version for large problems. Experiments on simulated and real-world tasks show that our method is more robust than standard two-step training and often gives better predictions.