Timezone: »
We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers. Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability. In order to obey this coverage property, however, conformal sets can become inundated with noisy candidates---which can render them unhelpful in practice. This is particularly relevant to practical applications where there is a limited budget, and the cost (monetary or otherwise) associated with false positives is non-negligible. We propose to trade coverage for a notion of precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false positives) is bounded according to a user-specified tolerance. Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true positive rate) that allows for any number of true answers for a given query (including zero). We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry.
Author Information
Adam Fisch (MIT)
Tal Schuster (Google)
Tommi Jaakkola (MIT)
Regina Barzilay (MIT CSAIL)

Regina Barzilay is an Israeli-American computer scientist. She is a professor at the Massachusetts Institute of Technology and a faculty lead for artificial intelligence at the MIT Jameel Clinic. Her research interests are in natural language processing and applications of deep learning to chemistry and oncology.
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: Conformal Prediction Sets with Limited False Positives »
Wed. Jul 20th 08:55 -- 09:00 PM Room Room 318 - 320
More from the Same Authors
-
2023 Poster: PFGM++: Unlocking the Potential of Physics-Inspired Generative Models »
Yilun Xu · Ziming Liu · Yonglong Tian · Shangyuan Tong · Max Tegmark · Tommi Jaakkola -
2023 Poster: Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models »
Guanhua Zhang · Jiabao Ji · Yang Zhang · Mo Yu · Tommi Jaakkola · Shiyu Chang -
2023 Poster: SE(3) diffusion model with application to protein backbone generation »
Jason Yim · Brian Trippe · Valentin De Bortoli · Emile Mathieu · Arnaud Doucet · Regina Barzilay · Tommi Jaakkola -
2022 : Spotlight Presentations »
Adrian Weller · Osbert Bastani · Jake Snell · Tal Schuster · Stephen Bates · Zhendong Wang · Margaux Zaffran · Danielle Rasooly · Varun Babbar -
2022 : Confident Adaptive Language Modeling »
Tal Schuster · Adam Fisch · Jai Gupta · Mostafa Dehghani · Dara Bahri · Vinh Tran · Yi Tay · Don Metzler -
2022 Poster: Learning Stable Classifiers by Transferring Unstable Features »
Yujia Bao · Shiyu Chang · Regina Barzilay -
2022 Poster: Antibody-Antigen Docking and Design via Hierarchical Structure Refinement »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2022 Spotlight: Learning Stable Classifiers by Transferring Unstable Features »
Yujia Bao · Shiyu Chang · Regina Barzilay -
2022 Spotlight: Antibody-Antigen Docking and Design via Hierarchical Structure Refinement »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2022 Poster: EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction »
Hannes Stärk · Octavian Ganea · Lagnajit Pattanaik · Regina Barzilay · Tommi Jaakkola -
2022 Spotlight: EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction »
Hannes Stärk · Octavian Ganea · Lagnajit Pattanaik · Regina Barzilay · Tommi Jaakkola -
2022 Invited Talk: Solving the Right Problems: Making ML Models Relevant to Healthcare and the Life Sciences »
Regina Barzilay -
2021 Poster: Few-Shot Conformal Prediction with Auxiliary Tasks »
Adam Fisch · Tal Schuster · Tommi Jaakkola · Regina Barzilay -
2021 Poster: Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers »
Yujia Bao · Shiyu Chang · Regina Barzilay -
2021 Spotlight: Few-Shot Conformal Prediction with Auxiliary Tasks »
Adam Fisch · Tal Schuster · Tommi Jaakkola · Regina Barzilay -
2021 Spotlight: Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers »
Yujia Bao · Shiyu Chang · Regina Barzilay -
2021 Poster: Information Obfuscation of Graph Neural Networks »
Peiyuan Liao · Han Zhao · Keyulu Xu · Tommi Jaakkola · Geoff Gordon · Stefanie Jegelka · Ruslan Salakhutdinov -
2021 Spotlight: Information Obfuscation of Graph Neural Networks »
Peiyuan Liao · Han Zhao · Keyulu Xu · Tommi Jaakkola · Geoff Gordon · Stefanie Jegelka · Ruslan Salakhutdinov -
2021 Poster: Learning Task Informed Abstractions »
Xiang Fu · Ge Yang · Pulkit Agrawal · Tommi Jaakkola -
2021 Spotlight: Learning Task Informed Abstractions »
Xiang Fu · Ge Yang · Pulkit Agrawal · Tommi Jaakkola -
2020 : Invited Talk: Tommi Jaakkola »
Tommi Jaakkola -
2020 Poster: Generalization and Representational Limits of Graph Neural Networks »
Vikas K Garg · Stefanie Jegelka · Tommi Jaakkola -
2020 Poster: Multi-Objective Molecule Generation using Interpretable Substructures »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2020 Poster: Educating Text Autoencoders: Latent Representation Guidance via Denoising »
Tianxiao Shen · Jonas Mueller · Regina Barzilay · Tommi Jaakkola -
2020 Poster: Invariant Rationalization »
Shiyu Chang · Yang Zhang · Mo Yu · Tommi Jaakkola -
2020 Poster: Predicting deliberative outcomes »
Vikas K Garg · Tommi Jaakkola -
2020 Poster: Hierarchical Generation of Molecular Graphs using Structural Motifs »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2020 Poster: Improving Molecular Design by Stochastic Iterative Target Augmentation »
Kevin Yang · Wengong Jin · Kyle Swanson · Regina Barzilay · Tommi Jaakkola -
2019 Poster: Functional Transparency for Structured Data: a Game-Theoretic Approach »
Guang-He Lee · Wengong Jin · David Alvarez-Melis · Tommi Jaakkola -
2019 Oral: Functional Transparency for Structured Data: a Game-Theoretic Approach »
Guang-He Lee · Wengong Jin · David Alvarez-Melis · Tommi Jaakkola -
2018 Poster: Junction Tree Variational Autoencoder for Molecular Graph Generation »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2018 Oral: Junction Tree Variational Autoencoder for Molecular Graph Generation »
Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2017 Poster: Learning Sleep Stages from Radio Signals: A Conditional Adversarial Architecture »
Mingmin Zhao · Shichao Yue · Dina Katabi · Tommi Jaakkola · Matt Bianchi -
2017 Talk: Learning Sleep Stages from Radio Signals: A Conditional Adversarial Architecture »
Mingmin Zhao · Shichao Yue · Dina Katabi · Tommi Jaakkola · Matt Bianchi -
2017 Poster: Sequence to Better Sequence: Continuous Revision of Combinatorial Structures »
Jonas Mueller · David Gifford · Tommi Jaakkola -
2017 Talk: Sequence to Better Sequence: Continuous Revision of Combinatorial Structures »
Jonas Mueller · David Gifford · Tommi Jaakkola -
2017 Poster: Deriving Neural Architectures from Sequence and Graph Kernels »
Tao Lei · Wengong Jin · Regina Barzilay · Tommi Jaakkola -
2017 Talk: Deriving Neural Architectures from Sequence and Graph Kernels »
Tao Lei · Wengong Jin · Regina Barzilay · Tommi Jaakkola