Timezone: »

 
Spotlight
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
John Miller · Rohan Taori · Aditi Raghunathan · Shiori Sagawa · Pang Wei Koh · Vaishaal Shankar · Percy Liang · Yair Carmon · Ludwig Schmidt

Wed Jul 21 06:45 PM -- 06:50 PM (PDT) @
in Learning Theory 10 »

For machine learning systems to be reliable, we must understand their performance in unseen, out- of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of- distribution performance on variants of CIFAR- 10 & ImageNet, a synthetic pose estimation task derived from YCB objects, FMoW-WILDS satellite imagery classification, and wildlife classification in iWildCam-WILDS. The correlation holds across model architectures, hyperparameters, training set size, and training duration, and is more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

Keywords: Models of Learning and Generalization

Author Information

John Miller (University of California, Berkeley)
Rohan Taori (Stanford University)
Aditi Raghunathan (Stanford)
Shiori Sagawa (Stanford University)
Pang Wei Koh (Stanford University)
Vaishaal Shankar (UC Berkeley)
Percy Liang (Stanford University)
Yair Carmon (Tel Aviv University)
Ludwig Schmidt (Toyota Research Institute)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors