Deep learning approaches are increasingly used in healthcare due to their seemingly remarkable performance. However, they can be notoriously brittle, often with little ability to generalize outside their training data. Using real life examples from ophthalmology, oncology and radiology, we will first discuss practical examples of distribution shifts. We will then highlight how even seemingly subtle distribution shifts can lead to catastrophic failures of models. We will highlight the need for constant vigilance of the input data and better metrics to quantify distribution shifts. We will conclude with a plea to the ICML/PODS community to work with clinical community on this critically important topic.