Timezone: »
Multiple lines of evidence suggest that predictive models may benefit from algorithmic triage. Under algorithmic triage, a predictive model does not predict all instances but instead defers some of them to human experts. However, the interplay between the prediction accuracy of the model and the human experts under algorithmic triage is not well understood. In this work, we start by formally characterizing under which circumstances a predictive model may benefit from algorithmic triage. In doing so, we also demonstrate that models trained for full automation may be suboptimal under triage. Then, given any model and the desired level of triage, we show that the optimal triage policy is a deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on a per-instance level. Building upon these results, we introduce a practical gradient-based algorithm that is guaranteed to find a sequence of predictive models and triage policies of increasing performance. Experiments on a wide variety of supervised learning tasks using synthetic and real data from two important applications---content moderation and scientific discovery---illustrate our theoretical results and show that the models and triage policies provided by our algorithm outperform those provided by several competitive baselines.
Author Information
Nastaran Okati (Max Planck Institute for Software Systems)
Abir De (IIT Bombay)
Manuel Gomez-Rodriguez (MPI-SWS)

Manuel Gomez Rodriguez is a faculty at Max Planck Institute for Software Systems. Manuel develops human-centric machine learning models and algorithms for the analysis, modeling and control of social, information and networked systems. He has received several recognitions for his research, including an outstanding paper award at NeurIPS’13 and a best research paper honorable mention at KDD’10 and WWW’17. He has served as track chair for FAT* 2020 and as area chair for every major conference in machine learning, data mining and the Web. Manuel has co-authored over 50 publications in top-tier conferences (NeurIPS, ICML, WWW, KDD, WSDM, AAAI) and journals (PNAS, Nature Communications, JMLR, PLOS Computational Biology). Manuel holds a BS in Electrical Engineering from Carlos III University, a MS and PhD in Electrical Engineering from Stanford University, and has received postdoctoral training at the Max Planck Institute for Intelligent Systems.
More from the Same Authors
-
2021 : Learning to Switch Among Agents in a Team »
Manuel Gomez-Rodriguez · Vahid Balazadeh Meresht -
2021 : Counterfactual Explanations in Sequential Decision Making Under Uncertainty »
Stratis Tsirtsis · Abir De · Manuel Gomez-Rodriguez -
2023 : Finding Counterfactually Optimal Action Sequences in Continuous State Spaces »
Stratis Tsirtsis · Manuel Gomez-Rodriguez -
2023 : Designing Decision Support Systems Using Counterfactual Prediction Sets »
Eleni Straitouri · Manuel Gomez-Rodriguez -
2023 : Human-Aligned Calibration for AI-Assisted Decision Making »
Nina Corvelo Benz · Manuel Gomez-Rodriguez -
2023 Workshop: “Could it have been different?” Counterfactuals in Minds and Machines »
Nina Corvelo Benz · Ricardo Dominguez-Olmedo · Manuel Gomez-Rodriguez · Thorsten Joachims · Amir-Hossein Karimi · Stratis Tsirtsis · Isabel Valera · Sarah A Wu -
2023 Poster: Improving Expert Predictions with Conformal Prediction »
Eleni Straitouri · Luke Lequn Wang · Nastaran Okati · Manuel Gomez-Rodriguez -
2023 Poster: On the Within-Group Fairness of Screening Classifiers »
Nastaran Okati · Stratis Tsirtsis · Manuel Gomez-Rodriguez -
2022 Poster: VarScene: A Deep Generative Model for Realistic Scene Graph Synthesis »
Tathagat Verma · Abir De · Yateesh Agrawal · Vishwa Vinay · Soumen Chakrabarti -
2022 Poster: Improving Screening Processes via Calibrated Subset Selection »
Luke Lequn Wang · Thorsten Joachims · Manuel Gomez-Rodriguez -
2022 Spotlight: Improving Screening Processes via Calibrated Subset Selection »
Luke Lequn Wang · Thorsten Joachims · Manuel Gomez-Rodriguez -
2022 Spotlight: VarScene: A Deep Generative Model for Realistic Scene Graph Synthesis »
Tathagat Verma · Abir De · Yateesh Agrawal · Vishwa Vinay · Soumen Chakrabarti -
2021 : Poster »
Shiji Zhou · Nastaran Okati · Wichinpong Sinchaisri · Kim de Bie · Ana Lucic · Mina Khan · Ishaan Shah · JINGHUI LU · Andreas Kirsch · Julius Frost · Ze Gong · Gokul Swamy · Ah Young Kim · Ahmed Baruwa · Ranganath Krishnan -
2021 : Differentiable learning Under Algorithmic Triage »
Manuel Gomez-Rodriguez -
2021 : Introduction by the Organizers »
Abir De · Rishabh Iyer · Ganesh Ramakrishnan · Jeff Bilmes -
2021 Workshop: Subset Selection in Machine Learning: From Theory to Applications »
Rishabh Iyer · Abir De · Ganesh Ramakrishnan · Jeff Bilmes -
2021 Workshop: ICML Workshop on Algorithmic Recourse »
Stratis Tsirtsis · Amir-Hossein Karimi · Ana Lucic · Manuel Gomez-Rodriguez · Isabel Valera · Hima Lakkaraju -
2021 Poster: GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training »
Krishnateja Killamsetty · Durga Sivasubramanian · Ganesh Ramakrishnan · Abir De · Rishabh Iyer -
2021 Spotlight: GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training »
Krishnateja Killamsetty · Durga Sivasubramanian · Ganesh Ramakrishnan · Abir De · Rishabh Iyer -
2021 Poster: Training Data Subset Selection for Regression with Controlled Generalization Error »
Durga Sivasubramanian · Rishabh Iyer · Ganesh Ramakrishnan · Abir De -
2021 Spotlight: Training Data Subset Selection for Regression with Controlled Generalization Error »
Durga Sivasubramanian · Rishabh Iyer · Ganesh Ramakrishnan · Abir De -
2018 Tutorial: Learning with Temporal Point Processes »
Manuel Gomez-Rodriguez · Isabel Valera