A Geometry-Based View of Mahalanobis OOD Detection

Out-of-distribution (OOD) detection is critical for reliable deployment of vision models, and Mahalanobis-based detectors remain strong baselines. However, their performance varies widely across modern pretrained representations, making it unclear which feature-space properties determine success or failure. Across diverse foundation-model backbones and Mahalanobis variants, we show that performance is highly representation-dependent and can shift substantially with pretraining data and fine-tuning. We link this variability to in-distribution geometry and identify a two-term ID summary that consistently tracks Mahalanobis OOD behavior across detectors: within-class spectral structure and local intrinsic dimensionality. Finally, we introduce radially scaled $\ell_2$ normalization, $\phi_\beta(z)=z/\|z\|^\beta$, a direction-preserving transformation that changes feature radii and exposes a different ID geometry to the same quadratic detector. Selecting $\beta$ from ID-only geometry signals generally outperforms fixed normalization baselines.

When an image-recognition model is deployed in the real world, it can encounter images unlike anything it saw during training. A reliable system should notice these unfamiliar inputs instead of making confident predictions on them. One common way to do this checks whether a new image looks close to the training examples inside the model’s internal feature space, using a distance measure called the Mahalanobis distance. We find that this simple detector can work very well for some modern vision models, but poorly for others, even when the models appear similar. To understand why, we study the geometry of these internal feature spaces across many pretrained vision models. We identify two measurable properties of the training features that help explain when Mahalanobis-based detection succeeds: how many local directions the data uses, and how concentrated each class is along its main directions. We then introduce a simple normalization that adjusts feature lengths while keeping their directions fixed, allowing the detector to better match the geometry of each model. This provides an easy, training-free way to tune Mahalanobis detectors using only in-distribution data, improving reliability without needing examples of every possible unfamiliar input.