Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation

Preference-based reward modeling is a core component of RLHF and DPO pipelines. In practice, the humans providing preference feedback are rarely an i.i.d. sample: recruitment and exposure often follow social, institutional, or spatial structure, inducing non-uniform inclusion probabilities that correlate with graph centrality. We formalize preference learning with *network-sampled* annotators and show that identity-agnostic scalar reward modeling implicitly represents an inclusion-weighted welfare, over-representing structurally central communities when the inclusion distribution $q$ differs from a designer-chosen target weighting $\pi$. We propose Graph-Preference Learning, which combines (i) a graph-personalized reward model that shares statistical strength across neighboring annotators and (ii) graph-balanced aggregation that computes stabilized importance weights to target $\pi$. Our analysis characterizes the induced welfare represented by the learned aggregate reward and bounds its deviation from the target in terms of weight mismatch, reward-model approximation, and finite-sample effects. Experiments on synthetic graphs and a *semi-synthetic* case study on the LMArena preference dataset, where biased inclusion is *induced* via graph-based sampling, demonstrate up to 62% reduction in target-welfare recovery error and 17% reduction in cross-language performance gaps under biased inclusion.

AI systems such as chatbots are often improved using feedback from people. A common approach is to show people two possible answers from an AI system and ask which one they prefer. The system then learns from these choices so that it can produce answers that better match human expectations. However, this process assumes that the people giving feedback fairly represent the wider population. In practice, this is often not true. Some people or communities may be easier to reach, more active online, or more connected to the platforms where feedback is collected. Their opinions may therefore have more influence than those of less visible groups. This paper studies how such imbalance can affect AI training when feedback providers are connected through social, institutional, geographic, or platform-based networks. We show that standard methods may unintentionally learn preferences that reflect the most represented or most connected groups, rather than the intended population as a whole. To address this, we propose Graph-Preference Learning, a method that uses information about relationships among feedback providers and adjusts how much each person’s feedback contributes. This helps the learned preference model better reflect a chosen target population. Our experiments, using both simulated networks and language-model preference data, show that the proposed method can more accurately recover the intended population preferences and reduce performance differences across language groups. Overall, this work aims to make human feedback for AI systems more representative, transparent, and fair.