RefChess: Training-Free Contextual Search for Zero-Shot Referring Image Segmentation

Recent advances in zero-shot referring image segmentation (RIS), driven by foundation models such as SAM and CLIP, have improved cross-modal alignment between visual regions and natural language expressions. Nevertheless, selecting the correct segmentation proposal remains challenging, as existing methods typically score proposals independently and can be distracted by visually similar candidates that partially satisfy the expression. To address this limitation, we propose RefChess, a training-free contextual search framework for robust proposal selection. Instead of treating proposal selection as a single-step ranking problem, RefChess evaluates candidate masks under sampled distractor contexts and uses Monte-Carlo Tree Search as a budgeted mechanism to explore the combinatorial space of contextual perturbations. The search is guided by a stability-aware reward that integrates language decomposition, vision--language similarity, object-centric cues, and spatial guidance signals. Experiments on standard RIS benchmarks show that RefChess consistently improves robustness and referring segmentation performance without task-specific training. Code is available at \url{https://github.com/Tongshiyan/RefChess}.

When people describe an object in an image using language, such as “the man on the far right” or “the black motorcycle behind the blue motorcycle,” an AI system needs to find exactly the object being referred to and outline it accurately. This is useful for visual search, human–robot interaction, assistive tools, and other applications where machines must understand images through natural language. This paper introduces RefChess, a method that helps AI choose the correct object in an image without requiring extra task-specific training. Existing methods often examine each possible object separately and pick the one that seems to best match the text. However, in crowded or confusing scenes, several objects may look similar or partially match the description, which can lead to mistakes. RefChess instead compares possible objects under different surrounding contexts, checking whether a candidate remains reliable even when similar distracting objects are considered. By combining information from the language description, the visual appearance of each candidate object, and simple spatial cues such as position or nearby objects, RefChess makes more stable decisions. Experiments on standard referring image segmentation benchmarks show that it improves the accuracy and robustness of selecting the intended object, while still avoiding the need for additional labeled training data. This makes language-guided image segmentation more practical and reliable in complex real-world scenes.