Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints

As acquiring reliable ground-truth labels is usually costly, or infeasible, crowdsourcing and aggregation of noisy human annotations is a common alternative. Aggregating subjective labels, though, may amplify individual biases, particularly regarding sensitive features, raising fairness concerns. Nonetheless, fairness in crowdsourced aggregation remains largely unexplored, with no existing convergence guarantees and only limited post-processing approaches for enforcing $\varepsilon$-fairness under demographic parity. We address this gap by analyzing the fairness of crowdsourced aggregation methods within the $\varepsilon$-fairness framework, for Majority Vote and Optimal Bayesian aggregation. In the small-crowd regime, we derive an upper bound on the fairness gap of Majority Vote in terms of the fairness gaps of the individual annotators. We further show that the fairness gap of the aggregated consensus converges exponentially fast to that of the ground-truth under interpretable conditions. Since ground-truth itself may still be unfair, we generalize a state-of-the-art multiclass fairness post-processing algorithm from the continuous to the discrete setting, which enforces strict demographic parity constraints on any aggregation rule. Experiments on synthetic and real datasets demonstrate the effectiveness of our approach and corroborate the theoretical insights.

Crowdsourced datasets are often built by asking many people to label the same examples, but human annotators can make systematic errors that depend on sensitive attributes such as race, gender, or whether a text mentions a protected group. Standard aggregation rules such as majority vote may preserve or even amplify these biases, especially when only a small number of annotators label each item. We study when common crowdsourcing methods are fair and show theoretically how their demographic-parity gap behaves as the number and quality of annotators changes. We then introduce FairCrowd, a post-processing method that can be applied on top of any label aggregation rule to enforce a user-specified fairness constraint. Unlike previous approaches, FairCrowd is designed for discrete crowdsourced labels and comes with formal guarantees. Experiments on synthetic data, Crowd Judgment, and Jigsaw Toxicity show that FairCrowd improves the fairness–accuracy trade-off compared with existing fair truth-discovery methods. The method is also computationally efficient, making it practical for real crowdsourcing pipelines.