A Causal Graphical Framework for Understanding Stability to Dataset Shifts

Growing interest in the external validity of prediction models has produced many methods for finding predictive distributions that are invariant to dataset shifts and can be used for prediction in new, unseen environments. However, these methods consider different types of shifts and have been developed under disparate frameworks, making it difficult to theoretically analyze how solutions differ with respect to stability and accuracy. Taking a causal graphical view, in this talk I will discuss three graphical operators for removing unstable parts of the DGP that correspond to three types of stable distributions. This clarifies the relationship between the types of “invariances” sought by many existing methods. Then, using an example from healthcare, I will demonstrate the tradeoff between minimax and average performance, highlighting the need for model developers to carefully determine when and how they achieve invariance.