Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

Large language models (LLMs) increasingly mediate decisions that affect what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench---the first expert-validated, scenario-based benchmark for human rights reasoning. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that frontier model accuracy scores range considerably across legal reasoning tasks ($\in$ 0.339-0.577), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical time.