Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation

Semi-supervised referring expression segmentation (SS-RES) aims to achieve precise pixel-level language grounding under limited annotation, yet suffers from limited supervision and unreliable pseudo-labels when exploiting unlabeled image–text pairs. In this work, we propose Learning to Label, a reinforced self-evolving framework (L2L) that casts pseudo-label construction as a learnable decision-making process. To build foundational understanding, we leverage a multimodal large language model to extract semantic–spatial priors, which are instantiated as initial soft segmentation proposals and elevated—together with textual cues—into learnable guidance signals that condition a hierarchical segmentation network. To ensure stable learning, a reinforced pseudo-label selection is further formulated as an exploratory decision process that adaptively rewards high-utility pixel-level supervision based on multimodal priors and model predictions. This reinforced self-evolving loop enables joint optimization of the segmentation model and pseudo-labels, progressively enhancing label reliability under sparse supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate improvements over existing methods, validating its effectiveness and generalization.

This paper aims to help computers find the exact object or region in an image described by a sentence, while using only a small amount of human annotation. This task is useful for image editing, robotics, and other interactive vision systems, but collecting detailed object masks for many image–sentence pairs is costly. We introduce Learning to Label, a method that learns from a few labeled examples and many unlabeled ones. It uses strong vision-language and segmentation models to provide rough guidance, but also learns which parts of that guidance are reliable. During training, the system gradually improves the automatically generated supervision instead of blindly trusting it. Experiments on widely used benchmarks show that our method works well under very limited labeled data and outperforms previous semi-supervised approaches, making language-guided image segmentation more data-efficient and practical.