Timezone: »
The fast-growing machine learning as a service industry has incubated many APIs for multi-label classification tasks such as OCR and multi-object recognition. The heterogeneity in those APIs’ price and performance, however, often forces users to make a choice between accuracy and expense. In this work, we propose FrugalMCT, a principled framework that jointly maximizes the accuracy while minimizes the expense by adaptively selecting the APIs to use for different data.. FrugalMCT combines different APIs’ predictions to improve accuracy, and selects which combination to use to respect expense constraint. Preliminary experiments using ML APIs from Google, Microsoft, and other providers for multi-label image classification show that FrugalMCT often achieves more than 50% cost reduction while matching the accuracy of the best single API.
Author Information
Lingjiao Chen (University of Wisconsin-Madison)
James Zou (Stanford University)
Matei Zaharia (Stanford and Databricks)
More from the Same Authors
-
2021 : Meaningfully Explaining a Model's Mistakes »
· Abubakar Abid · James Zou -
2021 : Meaningfully Explaining a Model's Mistakes »
Abubakar Abid · James Zou -
2021 : MetaDataset: A Dataset of Datasets for Evaluating Distribution Shifts and Training Conflicts »
Weixin Liang · James Zou · Weixin Liang -
2021 : Machine Learning API Shift Assessments: Change is Coming! »
Lingjiao Chen · James Zou · Matei Zaharia -
2021 : Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions »
Kailas Vodrahalli · James Zou -
2022 : On the nonlinear correlation of ML performance across data subpopulations »
Weixin Liang · Yining Mao · Yongchan Kwon · Xinyu Yang · James Zou -
2023 : Improve Model Inference Cost with Image Gridding »
Shreyas Krishnaswamy · Lisa Dunlap · Lingjiao Chen · Matei Zaharia · James Zou · Joseph Gonzalez -
2023 : Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value »
Yongchan Kwon · James Zou -
2022 : GSCLIP : A Framework for Explaining Distribution Shifts in Natural Language »
Zhiying Zhu · Weixin Liang · James Zou -
2022 : What Can Data-Centric AI Learn from Data Engineering? »
Matei Zaharia -
2022 : Evaluation of ML in Health/Science »
James Zou -
2022 : Data Sculpting: Interpretable Algorithm for End-to-End Cohort Selection »
Ruishan Liu · James Zou -
2022 : Data Budgeting for Machine Learning »
Weixin Liang · James Zou -
2022 Workshop: Knowledge Retrieval and Language Models »
Maithra Raghu · Urvashi Khandelwal · Chiyuan Zhang · Matei Zaharia · Alexander Rush -
2022 Poster: When and How Mixup Improves Calibration »
Linjun Zhang · Zhun Deng · Kenji Kawaguchi · James Zou -
2022 Poster: Efficient Online ML API Selection for Multi-Label Classification Tasks »
Lingjiao Chen · Matei Zaharia · James Zou -
2022 Poster: Improving Out-of-Distribution Robustness via Selective Augmentation »
Huaxiu Yao · Yu Wang · Sai Li · Linjun Zhang · Weixin Liang · James Zou · Chelsea Finn -
2022 Spotlight: Efficient Online ML API Selection for Multi-Label Classification Tasks »
Lingjiao Chen · Matei Zaharia · James Zou -
2022 Spotlight: Improving Out-of-Distribution Robustness via Selective Augmentation »
Huaxiu Yao · Yu Wang · Sai Li · Linjun Zhang · Weixin Liang · James Zou · Chelsea Finn -
2022 Spotlight: When and How Mixup Improves Calibration »
Linjun Zhang · Zhun Deng · Kenji Kawaguchi · James Zou -
2021 Poster: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Spotlight: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Poster: Memory-Efficient Pipeline-Parallel DNN Training »
Deepak Narayanan · Amar Phanishayee · Kaiyu Shi · Xie Chen · Matei Zaharia -
2021 Spotlight: Memory-Efficient Pipeline-Parallel DNN Training »
Deepak Narayanan · Amar Phanishayee · Kaiyu Shi · Xie Chen · Matei Zaharia -
2021 Poster: How to Learn when Data Reacts to Your Model: Performative Gradient Descent »
Zachary Izzo · Lexing Ying · James Zou -
2021 Spotlight: How to Learn when Data Reacts to Your Model: Performative Gradient Descent »
Zachary Izzo · Lexing Ying · James Zou -
2020 Poster: A Distributional Framework For Data Valuation »
Amirata Ghorbani · Michael Kim · James Zou -
2019 Poster: Concrete Autoencoders: Differentiable Feature Selection and Reconstruction »
Muhammed Fatih Balın · Abubakar Abid · James Zou -
2019 Poster: Discovering Conditionally Salient Features with Statistical Guarantees »
Jaime Roquero Gimenez · James Zou -
2019 Poster: LIT: Learned Intermediate Representation Training for Model Compression »
Animesh Koratana · Daniel Kang · Peter Bailis · Matei Zaharia -
2019 Oral: Discovering Conditionally Salient Features with Statistical Guarantees »
Jaime Roquero Gimenez · James Zou -
2019 Oral: LIT: Learned Intermediate Representation Training for Model Compression »
Animesh Koratana · Daniel Kang · Peter Bailis · Matei Zaharia -
2019 Oral: Concrete Autoencoders: Differentiable Feature Selection and Reconstruction »
Muhammed Fatih Balın · Abubakar Abid · James Zou -
2019 Poster: Data Shapley: Equitable Valuation of Data for Machine Learning »
Amirata Ghorbani · James Zou -
2019 Oral: Data Shapley: Equitable Valuation of Data for Machine Learning »
Amirata Ghorbani · James Zou -
2018 Poster: DRACO: Byzantine-resilient Distributed Training via Redundant Gradients »
Lingjiao Chen · Hongyi Wang · Zachary Charles · Dimitris Papailiopoulos -
2018 Oral: DRACO: Byzantine-resilient Distributed Training via Redundant Gradients »
Lingjiao Chen · Hongyi Wang · Zachary Charles · Dimitris Papailiopoulos -
2018 Poster: Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training »
Xi Wu · Wooyeong Jang · Jiefeng Chen · Lingjiao Chen · Somesh Jha -
2018 Oral: Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training »
Xi Wu · Wooyeong Jang · Jiefeng Chen · Lingjiao Chen · Somesh Jha -
2018 Poster: CoVeR: Learning Covariate-Specific Vector Representations with Tensor Decompositions »
Kevin Tian · Teng Zhang · James Zou -
2018 Oral: CoVeR: Learning Covariate-Specific Vector Representations with Tensor Decompositions »
Kevin Tian · Teng Zhang · James Zou