Timezone: »

 
Poster
Do ImageNet Classifiers Generalize to ImageNet?
Benjamin Recht · Rebecca Roelofs · Ludwig Schmidt · Vaishaal Shankar

Wed Jun 12 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #141
in Posters Wed »

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

Keywords: Robust Statistics and Machine Learning

Author Information

Benjamin Recht (Berkeley)

Benjamin Recht is an Associate Professor in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. Ben's research group studies the theory and practice of optimization algorithms with a focus on applications in machine learning, data analysis, and controls. Ben is the recipient of a Presidential Early Career Awards for Scientists and Engineers, an Alfred P. Sloan Research Fellowship, the 2012 SIAM/MOS Lagrange Prize in Continuous Optimization, the 2014 Jamon Prize, the 2015 William O. Baker Award for Initiatives in Research, and the 2017 NIPS Test of Time Award.

Rebecca Roelofs (University of California Berkeley)
Ludwig Schmidt (University of California, Berkeley)
Vaishaal Shankar (UC Berkeley)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors