Timezone: »
Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, (α-Precision, β-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.
Author Information
Ahmed Alaa (UCLA)
Boris van Breugel (University of Cambridge)
Evgeny S. Saveliev (University of Cambridge)
Mihaela van der Schaar (University of Cambridge and UCLA)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models »
Tue. Jul 19th through Wed the 20th Room Hall E #218
More from the Same Authors
-
2021 : Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators »
Alicia Curth · Mihaela van der Schaar -
2023 : Keynote 2: Mihaela van der Schaar (University of Cambridge) - Reality-Centric AI »
Mihaela van der Schaar -
2023 : Mihaela van der Schaar - Causal Deep Learning »
Mihaela van der Schaar -
2023 Poster: Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data »
Boris van Breugel · Zhaozhi Qian · Mihaela van der Schaar -
2023 Poster: Adaptive Identification of Populations with Treatment Benefit in Clinical Trials: Machine Learning Challenges and Solutions »
Alicia Curth · Alihan Hüyük · Mihaela van der Schaar -
2023 Poster: In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation »
Alicia Curth · Mihaela van der Schaar -
2023 Poster: Accounting For Informative Sampling When Learning to Forecast Treatment Outcomes Over Time »
Toon Vanderschueren · Alicia Curth · Wouter Verbeke · Mihaela van der Schaar -
2023 Poster: Learning Representations without Compositional Assumptions »
Tennison Liu · Jeroen Berrevoets · Zhaozhi Qian · Mihaela van der Schaar -
2023 Poster: Differentiable and Transportable Structure Learning »
Jeroen Berrevoets · Nabeel Seedat · Fergus Imrie · Mihaela van der Schaar -
2022 Poster: Inverse Contextual Bandits: Learning How Behavior Evolves over Time »
Alihan Hüyük · Daniel Jarrett · Mihaela van der Schaar -
2022 Poster: Data-SUITE: Data-centric identification of in-distribution incongruous examples »
Nabeel Seedat · Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations »
Nabeel Seedat · Fergus Imrie · Alexis Bellot · Zhaozhi Qian · Mihaela van der Schaar -
2022 Spotlight: Data-SUITE: Data-centric identification of in-distribution incongruous examples »
Nabeel Seedat · Jonathan Crabbé · Mihaela van der Schaar -
2022 Spotlight: Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations »
Nabeel Seedat · Fergus Imrie · Alexis Bellot · Zhaozhi Qian · Mihaela van der Schaar -
2022 Spotlight: Inverse Contextual Bandits: Learning How Behavior Evolves over Time »
Alihan Hüyük · Daniel Jarrett · Mihaela van der Schaar -
2022 Poster: Label-Free Explainability for Unsupervised Models »
Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: HyperImpute: Generalized Iterative Imputation with Automatic Model Selection »
Daniel Jarrett · Bogdan Cebere · Tennison Liu · Alicia Curth · Mihaela van der Schaar -
2022 Spotlight: HyperImpute: Generalized Iterative Imputation with Automatic Model Selection »
Daniel Jarrett · Bogdan Cebere · Tennison Liu · Alicia Curth · Mihaela van der Schaar -
2022 Spotlight: Label-Free Explainability for Unsupervised Models »
Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: Neural Laplace: Learning diverse classes of differential equations in the Laplace domain »
Samuel Holt · Zhaozhi Qian · Mihaela van der Schaar -
2022 Oral: Neural Laplace: Learning diverse classes of differential equations in the Laplace domain »
Samuel Holt · Zhaozhi Qian · Mihaela van der Schaar -
2022 Expo Talk Panel: Machine learning for drug discovery: Challenges and opportunities »
Mihaela van der Schaar · Jian Tang · A Martin Jones · Sébastien Lemieux · Thomas Watson -
2021 : Mihaela Van der Schaar: Time-series in healthcare: challenges and solutions »
Mihaela van der Schaar -
2021 Workshop: Self-Supervised Learning for Reasoning and Perception »
Pengtao Xie · Shanghang Zhang · Ishan Misra · Pulkit Agrawal · Katerina Fragkiadaki · Ruisi Zhang · Tassilo Klein · Asli Celikyilmaz · Mihaela van der Schaar · Eric Xing -
2021 : Quantitative epistemology: conceiving a new human-machine partnership »
Mihaela van der Schaar -
2021 Poster: Explaining Time Series Predictions with Dynamic Masks »
Jonathan Crabbé · Mihaela van der Schaar -
2021 Spotlight: Explaining Time Series Predictions with Dynamic Masks »
Jonathan Crabbé · Mihaela van der Schaar -
2021 Poster: Policy Analysis using Synthetic Controls in Continuous-Time »
Alexis Bellot · Mihaela van der Schaar -
2021 Spotlight: Policy Analysis using Synthetic Controls in Continuous-Time »
Alexis Bellot · Mihaela van der Schaar -
2021 Poster: Learning Queueing Policies for Organ Transplantation Allocation using Interpretable Counterfactual Survival Analysis »
Jeroen Berrevoets · Ahmed Alaa · Zhaozhi Qian · James Jordon · alexander gimson · Mihaela van der Schaar -
2021 Spotlight: Learning Queueing Policies for Organ Transplantation Allocation using Interpretable Counterfactual Survival Analysis »
Jeroen Berrevoets · Ahmed Alaa · Zhaozhi Qian · James Jordon · alexander gimson · Mihaela van der Schaar -
2021 Poster: Inverse Decision Modeling: Learning Interpretable Representations of Behavior »
Daniel Jarrett · Alihan Hüyük · Mihaela van der Schaar -
2021 Oral: Inverse Decision Modeling: Learning Interpretable Representations of Behavior »
Daniel Jarrett · Alihan Hüyük · Mihaela van der Schaar -
2021 Tutorial: Synthetic Healthcare Data Generation and Assessment: Challenges, Methods, and Impact on Machine Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2021 : Synthetic Healthcare Data Generation and Assessment: Challenges, Methods, and Impact on Machine Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2020 : Panel Discussion »
Neil Lawrence · Mihaela van der Schaar · Alex Smola · Valerio Perrone · Jack Parker-Holder · Zhengying Liu -
2020 : "Automated ML and its transformative impact on medicine and healthcare" by Mihaela van der Schaar »
Mihaela van der Schaar -
2020 : Invited Talk: Learning despite the unknown - missing data imputation in healthcare »
Mihaela van der Schaar -
2020 Poster: Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift »
Alexander Chan · Ahmed Alaa · Zhaozhi Qian · Mihaela van der Schaar -
2020 Poster: Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2020 Poster: Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounders »
Ioana Bica · Ahmed Alaa · Mihaela van der Schaar -
2020 Poster: Temporal Phenotyping using Deep Predictive Clustering of Disease Progression »
Changhee Lee · Mihaela van der Schaar -
2020 Poster: Learning for Dose Allocation in Adaptive Clinical Trials with Safety Constraints »
Cong Shen · Zhiyang Wang · Sofia Villar · Mihaela van der Schaar -
2020 Poster: Frequentist Uncertainty in Recurrent Neural Networks via Blockwise Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2020 Poster: Inverse Active Sensing: Modeling and Understanding Timely Decision-Making »
Daniel Jarrett · Mihaela van der Schaar -
2020 Tutorial: Machine Learning for Healthcare: Challenges, Methods, Frontiers »
Mihaela van der Schaar -
2019 Poster: Validating Causal Inference Models via Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2019 Oral: Validating Causal Inference Models via Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2018 Poster: AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Oral: AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Poster: Limits of Estimating Heterogeneous Treatment Effects: Guidelines for Practical Algorithm Design »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Oral: Limits of Estimating Heterogeneous Treatment Effects: Guidelines for Practical Algorithm Design »
Ahmed M. Alaa · Mihaela van der Schaar -
2017 Poster: Learning from Clinical Judgments: Semi-Markov-Modulated Marked Hawkes Processes for Risk Prognosis »
Ahmed M. Alaa · Scott B Hu · Mihaela van der Schaar -
2017 Talk: Learning from Clinical Judgments: Semi-Markov-Modulated Marked Hawkes Processes for Risk Prognosis »
Ahmed M. Alaa · Scott B Hu · Mihaela van der Schaar