Abstract: Detecting out-of-distribution (OOD) data with varying levels of semantic and covariate shifts with respect to the in-distribution (ID) is critical for the safe deployment of models. The goal is to design a detector that can accept meaningful variations of the ID data, while rejecting samples from OOD regimes. Such an objective can be realized by enforcing consistency with a scoring function (e.g., energy) and calibrating the detector to reject a curated set of OOD data (\textit{a.k.a} outlier exposure (OE)). However, OE methods require representative OOD datasets which are challenging to acquire in practice, hence the recent trend of designing OE-free detectors. In this paper, we find that controlled generalization to ID variations and exposure to diverse (synthetic) outliers are critical for improving OOD detection. Through empirical studies on the MedMNIST medical imaging benchmark, we demonstrate significant performance gains ($15\% - 35\%$ in AUROC) over existing OE-free, OOD detection approaches under both semantic and modality shifts.