UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model

The introduction of the Segment Anything Model (SAM) has paved the way for numerous semantic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of particular interest. However, the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification (UQ) approaches. This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation jointly respecting aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, a lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under-parameterised models, insufficient prompts or image ambiguities. Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accuracy and cost efficiency.

Artificial intelligence leads to strong results in various domains such as detecting and segmenting objects in images. For example, the famous Segment Anything Model (SAM) is able to segment arbitrary objects by clicking on a distinct point in the image. Unfortunately, humans cannot trust algorithms like SAM without further validations. Some predictions of SAM are possibly wrong because the choosen SAM model is not suitable to the image, the image contains ambiguous segments, or the prompt is badly choosen. To improve the validation process and to indicate how the prediction can be optimized, we introduce UncertainSAM, a simple method to estimate the uncertainty of SAM. Without much overhead, our method creates a few single valued indicators which estimate if a better model could improve the prediction, better prompts are necessary, or multiple valid results are existing in the image. Using our method, semi-automated pipelines can be improved by asking for user feedback in uncertain situations. We evaluate the method on various datasets and settings and provide easy-to-use code.