Timezone: »

 
Poster
Conformal Prediction Sets with Limited False Positives
Adam Fisch · Tal Schuster · Tommi Jaakkola · Regina Barzilay

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #629
in Poster Session 2 »

We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers. Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability. In order to obey this coverage property, however, conformal sets can become inundated with noisy candidates---which can render them unhelpful in practice. This is particularly relevant to practical applications where there is a limited budget, and the cost (monetary or otherwise) associated with false positives is non-negligible. We propose to trade coverage for a notion of precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false positives) is bounded according to a user-specified tolerance. Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true positive rate) that allows for any number of true answers for a given query (including zero). We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry.

Keywords: APP: Chemistry and Drug Discovery · APP: Computer Vision · APP: Language, Speech and Dialog · OPT: Optimization and Learning under Uncertainty · MISC: Everything Else · SA: Trustworthy Machine Learning

Author Information

Adam Fisch (MIT)
Tal Schuster (Google)
Tommi Jaakkola (MIT)
Regina Barzilay (MIT CSAIL)
Regina Barzilay

Regina Barzilay is an Israeli-American computer scientist. She is a professor at the Massachusetts Institute of Technology and a faculty lead for artificial intelligence at the MIT Jameel Clinic. Her research interests are in natural language processing and applications of deep learning to chemistry and oncology.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors