Quantifying Risk of Epistemic Harm from the Use of AI Surrogates in Social Science Research

Large Language Models (LLMs) are being used as ``AI surrogates'' for human participants in scientific studies, offering practical and ethical benefits in sensitive or potentially harmful settings, but also introducing risks of scientific invalidity and epistemic injustice. Invalidity arises when model-generated responses fail to faithfully capture the target phenomenon, while epistemic injustice arises when LLMs systematically misrepresent certain groups as sources of knowledge. In this work, we evaluate whether LLMs can serve as valid proxies for human subjects in studies of stereotype content. We compare responses from human annotators (n=193) and five open-source LLMs towards 50 intersectional identity groups and find systematic misalignment: models rate historically marginalized groups (e.g., Black, gay, and transgender women) more negatively, and historically privileged groups (e.g., White, cisgender, heterosexual men) more positively than human raters. To quantify algorithmic fidelity, we measure the Wasserstein distance between human and model responses and introduce the Fidelity Parity Ratio (FPR) to assess whether fidelity is comparable across subgroups. The mean Wasserstein distance between human and LLM responses is 1.5–2× larger than the within-human inter-trait baseline ($\approx$0.32), indicating distortion beyond the noise floor of the human sample. Moreover, fidelity varies systematically across groups, with FPR falling below the 4/5ths threshold across disability, race, nationality, language, and socio-economic status for all models. Our findings indicate that AI surrogates can misrepresent marginalized populations, risking scientific validity and equitable knowledge production.