PromptPilot: Game-Theoretic Multi-Agent Prompt Optimization for Segment Anything

Few shot segmentation with vision foundation models relies on high quality prompts to segment unseen categories from limited support annotations. Existing prompt construction methods depend on test-time adaptation, fixed heuristic sampling, or monolithic reinforcement learning, making it difficult to balance semantic consistency, spatial coverage, and prompt credit assignment. To address these limitations, PromptPilot is proposed as a hierarchical multi-agent reinforcement learning framework for point prompt optimization with frozen DINOv2 and SAM. Prompt construction is formulated as sequential decision making, where feature and physical agents propose complementary prompt modifications, and a manager agent selects actions using SAM feedback and local marginal contribution. PromptPilot functions as an inference-time optimization strategy without parameter updates. Extensive experiments demonstrate that the proposed game-theoretic approach improves segmentation performance and generalization, offering a principled solution for automated prompt engineering. The code is available at https://github.com/L-AILab/PromptPilot.

Image segmentation is the task of separating target objects from the background at the pixel level. In few shot segmentation, the goal is to segment new object categories when only a small number of labeled examples are available. Recent large vision models, such as SAM, can perform segmentation with user-provided prompts, but their performance strongly depends on whether these prompts are placed in useful locations. This paper introduces PromptPilot, a method that automatically optimizes point prompts for segmentation. Instead of retraining the large vision model, PromptPilot improves the prompts during inference. It treats prompt construction as a step-by-step decision process, where different agents focus on complementary goals: one agent emphasizes whether prompts are visually consistent with the target object, another considers whether prompts cover the object spatially, and a manager agent decides which prompt update should be applied. By using feedback from SAM and estimating how useful each prompt is, PromptPilot can select more reliable prompt configurations. Experiments on natural images, medical images, and video object segmentation show that PromptPilot improves segmentation performance and generalization over existing automatic prompt construction methods.