Timezone: »

 
LabelBench: A Comprehensive Framework for Benchmarking Label-Efficient Learning
Jifan Zhang · Yifang Chen · Gregory Canal · Stephen Mussmann · Yinglun Zhu · Simon Du · Kevin Jamieson · Robert Nowak

@
in DMLR Workshop: Data-centric Machine Learning Research »

Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be label-efficient: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates better label-efficiencies than previously reported in active learning. LabelBench's modular codebase will be open-sourced for the broader community to contribute label-efficient learning methods and benchmarks.

Author Information

Jifan Zhang (University of Wisconsin)
Yifang Chen (University of Washington)
Gregory Canal (University of Wisconsin-Madison)
Stephen Mussmann (University of Washington)
Stephen Mussmann

Steve is an IFDS postdoctoral fellow at the University of Washington Computer Science advised by Kevin Jamieson and Ludwig Schmidt. He received his PhD in 2021 from Stanford University Computer Science advised by Percy Liang. He focuses on machine learning research in data such as data selection and active learning.

Yinglun Zhu (University of California, Riverside)
Simon Du (University of Washington)
Kevin Jamieson (University of Washington)
Robert Nowak (University of Wisconsion-Madison)
Robert Nowak

Robert Nowak holds the Nosbusch Professorship in Engineering at the University of Wisconsin-Madison, where his research focuses on signal processing, machine learning, optimization, and statistics.

More from the Same Authors