Segment Anything with Robust Uncertainty-Accuracy Correlation

Despite strong zero-shot performance, SAM is unreliable under domain shift due to Mask-level Confidence Confusion (MCC), where a single IoU-based mask score fails to reflect pixel-wise reliability near boundaries. Motivated by the contrast between texture-biased shortcuts in neural networks and shape-centric processing in human vision, we model out-of-domain variation as appearance shifts and non-rigid deformations that jointly perturb images. We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts. RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry, and applies Uncertainty-Error Alignment to ensure uncertainty consistently highlights erroneous pixels even under adversarial perturbations. Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation.

Segment Anything Model (SAM, SAM2) segments an object instance from a set of click prompts and reports a single confidence number for the predicted mask. On images outside SAM2's training domain, such as X-rays or first-person video, the predictions become harder to interpret, and that confidence number stops reflecting whether each mask is correct, leaving users with no way to tell which pixels to trust. We made two changes. First, the decoder now outputs per-pixel uncertainty instead of one overall number. Second, two adversarial helper networks perturb each training image, shifting the object's appearance and bending its shape, searching for cases where the model is wrong but confident, or right but uncertain. Training on these hard cases teaches the model to keep uncertainty aligned with prediction error, an alignment that then generalizes to image domains never seen during training. Tested across 23 image domains spanning everyday objects, urban and indoor scenes, scientific imaging (X-rays, microscopy), and first-person video, RUAC is the first SAM variant whose uncertainty maps stay aligned with prediction errors at this scale. A radiologist can see which regions to trust instead of rechecking the whole prediction. The same training recipe could transfer to other foundation models.