DGN: Disagreement Graph Networks for Learning from Multiple Annotators

Crowdsourced label aggregation methods model each annotator independently, overlooking the collective structure of disagreements across the annotator pool---a structure that encodes which workers systematically corrupt the majority signal even when their mutual agreement is high. We propose DGN (Disagreement Graph Networks), which represents annotators as nodes in a weighted disagreement graph and uses eigenvector centrality as a reliability signal provably monotone with annotator error rate (Theorem 1). Our main contribution, DGN-S, requires no EM and no dataset-specific tuning. On benchmarks explicitly designed to stress-test correlated low-quality annotators---the regime our theory targets---DGN-S matches or exceeds EM-based state-of-the-art methods, while running $27\times$ faster than MACE at $M=100$ annotators (0.064s vs. 1.73s); in homogeneous annotator pools where graph structure carries weaker signal, accuracy degrades gracefully by at most 0.5 percentage points, consistent with our theoretical predictions. On the real Music genre benchmark, DGN-S equals MACE (80.1% vs. 80.0%) at a fraction of the runtime. For settings requiring interpretability, DGN-EM embeds the same graph as a Bayesian prior, yielding a label-free annotator reliability diagnostic (Spearman $\rho\approx 0.94$) with $O(M)$ parameters that remain stable in ultra-sparse regimes where $O(MK^2)$ confusion-matrix methods degrade.