Timezone: »

 
Poster
Active Learning for Decision-Making from Imbalanced Observational Data
Iiris Sundin · Peter Schulam · Eero Siivola · Aki Vehtari · Suchi Saria · Samuel Kaski

Tue Jun 11 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #239
in Posters Tue »
Machine learning can help personalized decision support by learning models to predict individual treatment effects (ITE). This work studies the reliability of prediction-based decision-making in a task of deciding which action $a$ to take for a target unit after observing its covariates $\tilde{x}$ and predicted outcomes $\hat{p}(\tilde{y} \mid \tilde{x}, a)$. An example case is personalized medicine and the decision of which treatment to give to a patient. A common problem when learning these models from observational data is imbalance, that is, difference in treated/control covariate distributions, which is known to increase the upper bound of the expected ITE estimation error. We propose to assess the decision-making reliability by estimating the ITE model's Type S error rate, which is the probability of the model inferring the sign of the treatment effect wrong. Furthermore, we use the estimated reliability as a criterion for active learning, in order to collect new (possibly expensive) observations, instead of making a forced choice based on unreliable predictions. We demonstrate the effectiveness of this decision-making aware active learning in two decision-making tasks: in simulated data with binary outcomes and in a medical dataset with synthetic and continuous treatment outcomes.
Keywords: Active Learning · Bayesian Methods

Author Information

Iiris Sundin (Aalto University)
Peter Schulam (Johns Hopkins University)
Eero Siivola (Aalto University)
Aki Vehtari (Aalto University)
Suchi Saria (Johns Hopkins University)
Samuel Kaski (Aalto University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors