SHERPA: Fine-tuning Segment Anything Models with Task-relevant Guidance

Segment Anything Models (SAMs) often struggle with certain specialized tasks. A common approach is to fine-tune models with specific task labels, but this often leads to overfitting, introduces model bias and significantly degrades their generalization ability. To overcome these challenges, we propose SHERPA, a novel framework that leverages a smaller SAM to guide the fine-tuning of a larger SAM via task-relevant features. Specifically, we first leverage the Fisher Ratio Separation (FRS) module to separate high task-relevant features and preserve the ability of the large SAM to perform other general tasks. Then, the Guiding Feature Extraction (GFE) module is used to extract representative guiding features from the fine-tuned small SAMs. We leverage small SAMs tailored for specific tasks (including natural image segmentation, biomedical image segmentation, and video object segmentation) as guidance and then evaluate the SHERPA scheme to fine-tune larger SAM series models. Our experiments demonstrate that SHERPA enhances the retention of generalization ability across those diverse tasks, by up to 11.1\%, and improves specific task performance by up to 2.2\%.

Segment Anything Models (SAMs) are powerful artificial intelligence systems capable of identifying and outlining objects within any image. However, when researchers try to adapt these models to specialized tasks—such as medical imaging or tracing objects in videos—the models often overfit to the new data, causing them to "forget" their broad, general-purpose abilities. To overcome this challenge, we introduce SHERPA, a novel approach that uses a smaller, highly focused AI model to act as a "guide" for the larger foundation model. Instead of allowing the large model to change completely during training, SHERPA isolates and protects its general knowledge while safely steering it to learn the new specialized task. Our experiments show that this collaborative approach effectively boosts the model's accuracy on custom tasks while significantly preserving its original, broad zero-shot capabilities. This makes advanced visual AI much more robust and adaptable for diverse real-world applications.