MA$^3$S: Model-Agnostic Active Annotation Strategy for Crowdsourcing

In crowdsourcing scenarios, to mitigate the impact of noisy labels assigned by non-expert workers, each instance is typically annotated multiple times by different workers. However, repeated annotation can introduce instance- or label-level redundancy, thereby inflating annotation costs. Despite its practical importance, research on repeated annotation strategies remains limited, and no existing strategy simultaneously avoids being offline, instance-unaware, and model-centric. In this paper, we propose a model-agnostic active annotation strategy, MA$^3$S, that addresses these limitations: (1) To reduce label redundancy caused by offline procedure, MA$^3$S estimates instance uncertainties with a margin-Beta distribution and updates them online as new labels arrive. (2) To prevent instance redundancy induced by instance-unaware designs, MA$^3$S constructs a nearest-neighbor graph to propagate instance uncertainties, reducing repeated annotations of similar instances. (3) To avoid being model-centric, MA$^3$S actively selects instances for annotation based solely on the estimated uncertainties, without relying on model feedback. Extensive experiments on synthetic and real-world datasets demonstrate that MA$^3$S consistently outperforms existing annotation strategies.

In crowdsourcing tasks, labels are often provided by non-experts, so each data instance is usually annotated multiple times to improve reliability. However, repeatedly collecting labels can lead to unnecessary duplication at both the instance and label levels, increasing annotation costs without always bringing additional benefits. Although this issue is important in practice, existing strategies for deciding when and what to annotate repeatedly are still limited. Many of them either rely on offline processing, ignore relationships between instances, or depend heavily on specific machine learning models. To address these problems, we propose a general annotation strategy called MA$^3$S. It dynamically updates how uncertain each instance is as new labels are collected, reducing redundant labeling caused by offline decisions. It also considers similarities between instances, so that similar instances are less likely to be annotated repeatedly. In addition, MA$^3$S does not depend on any specific model, making it more flexible in different applications. Experiments on both synthetic and real-world datasets show that MA$^3$S can more effectively use annotation effort and achieve better overall performance compared to existing strategies.