Beyond Confidence: Exploiting Homogeneous Pattern for Semi-Supervised Semantic Segmentation

The critical challenge of semi-supervised semantic segmentation lies in how to fully exploit a large volume of unlabeled data to improve the model's generalization performance for robust segmentation. Existing methods mainly rely on confidence-based scoring functions in the prediction space to filter pseudo labels, which suffer from the inherent trade-off between true and false positive rates. In this paper, we carefully design an agent construction strategy to build clean sets of correct (positive) and incorrect (negative) pseudo labels, and propose the Agent Score function (AgScore) to measure the consensus between candidate pixels and these sets. In this way, AgScore takes a step further to capture homogeneous patterns in the embedding space, conditioned on clean positive/negative agents stemming from the prediction space, without sacrificing the merits of confidence score, yielding better trad-off. We provide theoretical analysis to understand the mechanism of AgScore, and demonstrate its effectiveness by integrating it into three semi-supervised segmentation frameworks on Pascal VOC, Cityscapes, and COCO datasets, showing consistent improvements across all data partitions.

Training AI to understand images often needs many labeled examples, which is costly. Semi-supervised learning reduces this need by using both labeled and unlabeled images. A key challenge is deciding which AI-generated “pseudo labels” are reliable. Most existing methods rely on the model’s confidence, which can be misleading. Our work introduces a new scoring method called AgScore. It looks at how similar each prediction is to trusted patterns in the model’s internal features, instead of just using confidence. This helps select better training signals from unlabeled data. AgScore improves learning across several datasets, making it easier to train accurate models with less labeled data.