Timezone: »
Recurrent neural networks (RNNs) are instrumental in modelling sequential and time-series data. Yet, when using RNNs to inform decision-making, predictions by themselves are not sufficient — we also need estimates of predictive uncertainty. Existing approaches for uncertainty quantification in RNNs are based predominantly on Bayesian methods; these are computationally prohibitive, and require major alterations to the RNN architecture and training. Capitalizing on ideas from classical jackknife resampling, we develop a frequentist alternative that: (a) does not interfere with model training or compromise its accuracy, (b) applies to any RNN architecture, and (c) provides theoretical coverage guarantees on the estimated uncertainty intervals. Our method derives predictive uncertainty from the variability of the (jackknife) sampling distribution of the RNN outputs, which is estimated by repeatedly deleting “blocks” of (temporally-correlated) training data, and collecting the predictions of the RNN re-trained on the remaining data. To avoid exhaustive re-training, we utilize influence functions to estimate the effect of removing training data blocks on the learned RNN parameters. Using data from a critical care setting, we demonstrate the utility of uncertainty quantification in sequential decision-making.
Author Information
Ahmed Alaa (UCLA)
Mihaela van der Schaar (University of Cambridge and UCLA)
More from the Same Authors
-
2021 : Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators »
Alicia Curth · Mihaela van der Schaar -
2023 : Keynote 2: Mihaela van der Schaar (University of Cambridge) - Reality-Centric AI »
Mihaela van der Schaar -
2023 : Mihaela van der Schaar - Causal Deep Learning »
Mihaela van der Schaar -
2023 Poster: Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data »
Boris van Breugel · Zhaozhi Qian · Mihaela van der Schaar -
2023 Poster: Adaptive Identification of Populations with Treatment Benefit in Clinical Trials: Machine Learning Challenges and Solutions »
Alicia Curth · Alihan Hüyük · Mihaela van der Schaar -
2023 Poster: In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation »
Alicia Curth · Mihaela van der Schaar -
2023 Poster: Accounting For Informative Sampling When Learning to Forecast Treatment Outcomes Over Time »
Toon Vanderschueren · Alicia Curth · Wouter Verbeke · Mihaela van der Schaar -
2023 Poster: Learning Representations without Compositional Assumptions »
Tennison Liu · Jeroen Berrevoets · Zhaozhi Qian · Mihaela van der Schaar -
2023 Poster: Differentiable and Transportable Structure Learning »
Jeroen Berrevoets · Nabeel Seedat · Fergus Imrie · Mihaela van der Schaar -
2022 Poster: Inverse Contextual Bandits: Learning How Behavior Evolves over Time »
Alihan Hüyük · Daniel Jarrett · Mihaela van der Schaar -
2022 Poster: Data-SUITE: Data-centric identification of in-distribution incongruous examples »
Nabeel Seedat · Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations »
Nabeel Seedat · Fergus Imrie · Alexis Bellot · Zhaozhi Qian · Mihaela van der Schaar -
2022 Spotlight: Data-SUITE: Data-centric identification of in-distribution incongruous examples »
Nabeel Seedat · Jonathan Crabbé · Mihaela van der Schaar -
2022 Spotlight: Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations »
Nabeel Seedat · Fergus Imrie · Alexis Bellot · Zhaozhi Qian · Mihaela van der Schaar -
2022 Spotlight: Inverse Contextual Bandits: Learning How Behavior Evolves over Time »
Alihan Hüyük · Daniel Jarrett · Mihaela van der Schaar -
2022 Poster: Label-Free Explainability for Unsupervised Models »
Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: HyperImpute: Generalized Iterative Imputation with Automatic Model Selection »
Daniel Jarrett · Bogdan Cebere · Tennison Liu · Alicia Curth · Mihaela van der Schaar -
2022 Spotlight: HyperImpute: Generalized Iterative Imputation with Automatic Model Selection »
Daniel Jarrett · Bogdan Cebere · Tennison Liu · Alicia Curth · Mihaela van der Schaar -
2022 Spotlight: Label-Free Explainability for Unsupervised Models »
Jonathan Crabbé · Mihaela van der Schaar -
2022 Poster: How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models »
Ahmed Alaa · Boris van Breugel · Evgeny S. Saveliev · Mihaela van der Schaar -
2022 Poster: Neural Laplace: Learning diverse classes of differential equations in the Laplace domain »
Samuel Holt · Zhaozhi Qian · Mihaela van der Schaar -
2022 Oral: Neural Laplace: Learning diverse classes of differential equations in the Laplace domain »
Samuel Holt · Zhaozhi Qian · Mihaela van der Schaar -
2022 Spotlight: How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models »
Ahmed Alaa · Boris van Breugel · Evgeny S. Saveliev · Mihaela van der Schaar -
2022 Expo Talk Panel: Machine learning for drug discovery: Challenges and opportunities »
Mihaela van der Schaar · Jian Tang · A Martin Jones · Sébastien Lemieux · Thomas Watson -
2021 : Mihaela Van der Schaar: Time-series in healthcare: challenges and solutions »
Mihaela van der Schaar -
2021 Workshop: Self-Supervised Learning for Reasoning and Perception »
Pengtao Xie · Shanghang Zhang · Ishan Misra · Pulkit Agrawal · Katerina Fragkiadaki · Ruisi Zhang · Tassilo Klein · Asli Celikyilmaz · Mihaela van der Schaar · Eric Xing -
2021 : Quantitative epistemology: conceiving a new human-machine partnership »
Mihaela van der Schaar -
2021 Poster: Explaining Time Series Predictions with Dynamic Masks »
Jonathan Crabbé · Mihaela van der Schaar -
2021 Spotlight: Explaining Time Series Predictions with Dynamic Masks »
Jonathan Crabbé · Mihaela van der Schaar -
2021 Poster: Policy Analysis using Synthetic Controls in Continuous-Time »
Alexis Bellot · Mihaela van der Schaar -
2021 Spotlight: Policy Analysis using Synthetic Controls in Continuous-Time »
Alexis Bellot · Mihaela van der Schaar -
2021 Poster: Learning Queueing Policies for Organ Transplantation Allocation using Interpretable Counterfactual Survival Analysis »
Jeroen Berrevoets · Ahmed Alaa · Zhaozhi Qian · James Jordon · alexander gimson · Mihaela van der Schaar -
2021 Spotlight: Learning Queueing Policies for Organ Transplantation Allocation using Interpretable Counterfactual Survival Analysis »
Jeroen Berrevoets · Ahmed Alaa · Zhaozhi Qian · James Jordon · alexander gimson · Mihaela van der Schaar -
2021 Poster: Inverse Decision Modeling: Learning Interpretable Representations of Behavior »
Daniel Jarrett · Alihan Hüyük · Mihaela van der Schaar -
2021 Oral: Inverse Decision Modeling: Learning Interpretable Representations of Behavior »
Daniel Jarrett · Alihan Hüyük · Mihaela van der Schaar -
2021 Tutorial: Synthetic Healthcare Data Generation and Assessment: Challenges, Methods, and Impact on Machine Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2021 : Synthetic Healthcare Data Generation and Assessment: Challenges, Methods, and Impact on Machine Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2020 : Panel Discussion »
Neil Lawrence · Mihaela van der Schaar · Alex Smola · Valerio Perrone · Jack Parker-Holder · Zhengying Liu -
2020 : "Automated ML and its transformative impact on medicine and healthcare" by Mihaela van der Schaar »
Mihaela van der Schaar -
2020 : Invited Talk: Learning despite the unknown - missing data imputation in healthcare »
Mihaela van der Schaar -
2020 Poster: Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift »
Alexander Chan · Ahmed Alaa · Zhaozhi Qian · Mihaela van der Schaar -
2020 Poster: Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2020 Poster: Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounders »
Ioana Bica · Ahmed Alaa · Mihaela van der Schaar -
2020 Poster: Temporal Phenotyping using Deep Predictive Clustering of Disease Progression »
Changhee Lee · Mihaela van der Schaar -
2020 Poster: Learning for Dose Allocation in Adaptive Clinical Trials with Safety Constraints »
Cong Shen · Zhiyang Wang · Sofia Villar · Mihaela van der Schaar -
2020 Poster: Inverse Active Sensing: Modeling and Understanding Timely Decision-Making »
Daniel Jarrett · Mihaela van der Schaar -
2020 Tutorial: Machine Learning for Healthcare: Challenges, Methods, Frontiers »
Mihaela van der Schaar -
2019 Poster: Validating Causal Inference Models via Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2019 Oral: Validating Causal Inference Models via Influence Functions »
Ahmed Alaa · Mihaela van der Schaar -
2018 Poster: AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Oral: AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Poster: Limits of Estimating Heterogeneous Treatment Effects: Guidelines for Practical Algorithm Design »
Ahmed M. Alaa · Mihaela van der Schaar -
2018 Oral: Limits of Estimating Heterogeneous Treatment Effects: Guidelines for Practical Algorithm Design »
Ahmed M. Alaa · Mihaela van der Schaar -
2017 Poster: Learning from Clinical Judgments: Semi-Markov-Modulated Marked Hawkes Processes for Risk Prognosis »
Ahmed M. Alaa · Scott B Hu · Mihaela van der Schaar -
2017 Talk: Learning from Clinical Judgments: Semi-Markov-Modulated Marked Hawkes Processes for Risk Prognosis »
Ahmed M. Alaa · Scott B Hu · Mihaela van der Schaar