Probably Approximately Correct Labels

Obtaining high-quality labeled datasets is often costly, requiring either human annotation or expensive experiments. In theory, powerful pre-trained AI models provide an opportunity to automatically label datasets and save costs. Unfortunately, these models come with no guarantees on their accuracy, making wholesale replacement of manual labeling impractical. In this work, we propose a method for leveraging pre-trained AI models to curate cost-effective and high-quality datasets. In particular, our approach results in probably approximately correct labels: with high probability, the overall labeling error is small. Our method is nonasymptotically valid under minimal assumptions on the dataset or the AI model being studied, and thus enables rigorous yet efficient dataset curation using modern AI models. We demonstrate the benefits of the methodology through text annotation with large language models, image labeling with pre-trained vision models, and protein folding analysis with AlphaFold.

Building good datasets usually requires manual human annotation or costly experiments. Modern AI models can do this labeling automatically and cheaply, but they make mistakes. Without a way to control those mistakes, we cannot trust their output. This paper introduces a method that uses pre-trained AI models to label data cheaply while still giving a mathematical guarantee that the total number of labeling errors stays small, with high confidence. The guarantee holds without needing strong assumptions about the data or the AI model being used, so it works with any model the researcher has at their disposal. We demonstrate the benefits of the approach through text annotation with large language models, image labeling with pre-trained vision models, and protein folding analysis with AlphaFold.