Timezone: »
Statistical learning theory is at an inflection point enabled by recent advances in understanding and optimizing a wide range of metrics. Of particular interest are non-decomposable metrics such as the F-measure and the Jaccard measure which cannot be represented as a simple average over examples. Non-decomposability is the primary source of difficulty in theoretical analysis, and interestingly has led to two distinct settings and notions of consistency. In this manuscript we analyze both settings, from statistical and algorithmic points of view, to explore the connections and to highlight differences between them for a wide range of metrics. The analysis complements previous results on this topic, clarifies common confusions around both settings, and provides guidance to the theory and practice of binary classification with complex metrics.
Author Information
Krzysztof Dembczynski (Poznan University of Technology)
Krzysztof Dembczyński is an assistant professor at Poznań University of Technology. He received his B.Sc., M.Sc., and Ph.D. degrees in computer science from the same university. As a post-doctoral researcher he spent two years from 2009 to 2011 in the Knowledge Engineering & Bioinformatics Lab at Marburg University, Germany. His articles have been published at the main conferences (ECML,ICML, NIPS) and in the leading journals (JMLR, MLJ, DAMI) in the field of machine learning. As a co-author he won the best paper award at the European Conference on Artificial Intelligence 2012 and at the Asian Conference on Machine Learning 2015. He also gave a tutorial on multi-target prediction problems at the International Conference on Machine Learning 2013 and at Algorithmic Learning Theory/Discovery Science 2013. He serves as a member of the program committees of major conferences in the field of artificial intelligence (ICML, NIPS, IJCAI, AAAI, KDD) and as a reviewer for several international journals (MLJ, DAMI, JMLR). He is a laureate of a prestigious scholarship in the HOMING PLUS programme awarded by the Foundation for Polish Science (2012– 2014). He was also receiving a stipend for outstanding young scientists funded by the Polish Ministry of Science and Higher Education (2011–2013).
Wojciech Kotlowski (Poznan University of Technology)
Sanmi Koyejo (University of Illinois at Urbana-Champaign)

Sanmi (Oluwasanmi) Koyejo is an Assistant Professor in the Department of Computer Science at Stanford University. Koyejo was previously an Associate Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. Koyejo's research interests are in developing the principles and practice of trustworthy machine learning, focusing on applications to neuroscience and healthcare. Koyejo completed a Ph.D. in Electrical Engineering at the University of Texas at Austin, advised by Joydeep Ghosh, and postdoctoral research at Stanford University with Russell A. Poldrack and Pradeep Ravikumar. Koyejo has been the recipient of several awards, including a best paper award from the conference on uncertainty in artificial intelligence, a Skip Ellis Early Career Award, a Sloan Fellowship, a Terman faculty fellowship, an NSF CAREER award, a Kavli Fellowship, an IJCAI early career spotlight, and a trainee award from the Organization for Human Brain Mapping. Koyejo spends time at Google as a part of the Brain team, serves on the Neural Information Processing Systems Foundation Board, the Association for Health Learning and Inference Board, and as president of the Black in AI organization.
Nagarajan Natarajan (Microsoft Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Talk: Consistency Analysis for Binary Classification Revisited »
Tue. Aug 8th 04:42 -- 05:00 AM Room C4.8
More from the Same Authors
-
2022 : Adapting to Shifts in Latent Confounders via Observed Concepts and Proxies »
Matt Kusner · Ibrahim Alabdulmohsin · Stephen Pfohl · Olawale Salaudeen · Arthur Gretton · Sanmi Koyejo · Jessica Schrouff · Alexander D'Amour -
2023 : Layer-Wise Feedback Alignment is Conserved in Deep Neural Networks »
Zach Robertson · Sanmi Koyejo -
2023 : FACADE: A Framework for Adversarial Circuit Anomaly Detection and Evaluation »
Dhruv Pai · Andres Carranza · Rylan Schaeffer · Arnuv Tandon · Sanmi Koyejo -
2023 : Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Pre-trained on Formally Diverse Data »
Alycia Lee · Brando Miranda · Brando Miranda · Sanmi Koyejo -
2023 : Is Pre-training Truly Better Than Meta-Learning? »
Brando Miranda · Patrick Yu · Saumya Goyal · Yu-Xiong Wang · Sanmi Koyejo -
2023 : Leveraging Side Information for Communication-Efficient Federated Learning »
Berivan Isik · Francesco Pase · Deniz Gunduz · Sanmi Koyejo · Tsachy Weissman · Michele Zorzi -
2023 : Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting »
Rylan Schaeffer · Kateryna Pistunova · Samar Khanna · Sarthak Consul · Sanmi Koyejo -
2023 : GPT-Zip: Deep Compression of Finetuned Large Language Models »
Berivan Isik · Hermann Kumbong · Wanyi Ning · Xiaozhe Yao · Sanmi Koyejo · Ce Zhang -
2023 : Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Pre-trained on Formally Diverse Data »
Alycia Lee · Brando Miranda · Sanmi Koyejo -
2023 : Are Emergent Abilities of Large Language Models a Mirage? »
Rylan Schaeffer · Brando Miranda · Sanmi Koyejo -
2023 : Thomas: Learning to Explore Human Preference via Probabilistic Reward Model »
Sang Truong · Duc Nguyen · Tho Quan · Sanmi Koyejo -
2023 : On learning domain general predictors »
Sanmi Koyejo -
2023 : Deceptive Alignment Monitoring »
Andres Carranza · Dhruv Pai · Rylan Schaeffer · Arnuv Tandon · Sanmi Koyejo -
2023 : Vignettes on Pairwise-Feedback Mechanisms for Learning with Uncertain Preferences »
Sanmi Koyejo -
2023 Workshop: 2nd ICML Workshop on New Frontiers in Adversarial Machine Learning »
Sijia Liu · Pin-Yu Chen · Dongxiao Zhu · Eric Wong · Kathrin Grosse · Baharan Mirzasoleiman · Sanmi Koyejo -
2023 Panel: The Societal Impacts of AI »
Sanmi Koyejo · Samy Bengio · Ashia Wilson · Kirikowhai Mikaere · Joelle Pineau -
2023 Poster: Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions »
Boxiang Lyu · Zhe Feng · Zach Robertson · Sanmi Koyejo -
2022 Workshop: New Frontiers in Adversarial Machine Learning »
Sijia Liu · Pin-Yu Chen · Dongxiao Zhu · Eric Wong · Kathrin Grosse · Hima Lakkaraju · Sanmi Koyejo -
2022 Poster: Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization »
Xiaojun Xu · Yibo Zhang · Evelyn Ma · Hyun Ho Son · Sanmi Koyejo · Bo Li -
2022 Spotlight: Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization »
Xiaojun Xu · Yibo Zhang · Evelyn Ma · Hyun Ho Son · Sanmi Koyejo · Bo Li -
2021 Poster: Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability »
Kaizhao Liang · Yibo Zhang · Boxin Wang · Zhuolin Yang · Sanmi Koyejo · Bo Li -
2021 Spotlight: Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability »
Kaizhao Liang · Yibo Zhang · Boxin Wang · Zhuolin Yang · Sanmi Koyejo · Bo Li -
2021 Poster: Optimizing Black-box Metrics with Iterative Example Weighting »
Gaurush Hiranandani · Jatin Mathur · Harikrishna Narasimhan · Mahdi Milani Fard · Sanmi Koyejo -
2021 Spotlight: Optimizing Black-box Metrics with Iterative Example Weighting »
Gaurush Hiranandani · Jatin Mathur · Harikrishna Narasimhan · Mahdi Milani Fard · Sanmi Koyejo -
2020 : Discussion Panel »
Krzysztof Dembczynski · Prateek Jain · Alina Beygelzimer · Inderjit Dhillon · Anna Choromanska · Maryam Majzoubi · Yashoteja Prabhu · John Langford -
2020 Poster: On the consistency of top-k surrogate losses »
Forest Yang · Sanmi Koyejo -
2020 Poster: Optimization and Analysis of the pAp@k Metric for Recommender Systems »
Gaurush Hiranandani · Warut Vijitbenjaronk · Sanmi Koyejo · Prateek Jain -
2020 Poster: Zeno++: Robust Fully Asynchronous SGD »
Cong Xie · Sanmi Koyejo · Indranil Gupta -
2019 Poster: Partially Linear Additive Gaussian Graphical Models »
Sinong Geng · Minhao Yan · Mladen Kolar · Sanmi Koyejo -
2019 Poster: Adaptive Scale-Invariant Online Algorithms for Learning Linear Models »
Michal Kempka · Wojciech Kotlowski · Manfred K. Warmuth -
2019 Oral: Partially Linear Additive Gaussian Graphical Models »
Sinong Geng · Minhao Yan · Mladen Kolar · Sanmi Koyejo -
2019 Oral: Adaptive Scale-Invariant Online Algorithms for Learning Linear Models »
Michal Kempka · Wojciech Kotlowski · Manfred K. Warmuth -
2019 Poster: Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance »
Cong Xie · Sanmi Koyejo · Indranil Gupta -
2019 Oral: Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance »
Cong Xie · Sanmi Koyejo · Indranil Gupta -
2018 Poster: Binary Classification with Karmic, Threshold-Quasi-Concave Metrics »
Bowei Yan · Sanmi Koyejo · Kai Zhong · Pradeep Ravikumar -
2018 Oral: Binary Classification with Karmic, Threshold-Quasi-Concave Metrics »
Bowei Yan · Sanmi Koyejo · Kai Zhong · Pradeep Ravikumar -
2017 Poster: Active Heteroscedastic Regression »
Kamalika Chaudhuri · Prateek Jain · Nagarajan Natarajan -
2017 Talk: Active Heteroscedastic Regression »
Kamalika Chaudhuri · Prateek Jain · Nagarajan Natarajan