Beyond Procedure: Substantive Fairness in Conformal Prediction

Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with fairness in downstream decision-making remains underexplored. Moving beyond CP as a standalone operation (procedural fairness), we analyze the holistic decision-making pipeline to evaluate substantive fairness—the equity of downstream outcomes. Theoretically, we derive an upper bound that decomposes prediction-set size disparity into interpretable components, clarifying how label-clustered CP helps control method-driven contributions to unfairness. To facilitate scalable empirical analysis, we introduce an LLM-in-the-loop evaluator that approximates human assessment of substantive fairness across diverse modalities. Our experiments show that label-clustered CP often provides a favorable balance between utility and substantive fairness, while reducing set-size disparities in line with our theory. Finally, we empirically show that equalized set sizes, rather than coverage, strongly correlate with improved substantive fairness, enabling practitioners to design more fair CP systems. Our code is available at https://github.com/layer6ai-labs/llm-in-the-loop-conformal-fairness.

Conformal prediction (CP) helps an AI system express uncertainty by returning a set of possible answers, with a coverage guarantee that the correct answer is included most of the time. When CP sets are used as decision aids, our goal is substantive fairness: they should improve accuracy by similar amounts across sensitive groups, compared with decisions made without CP assistance. Yet a CP method can look fair statistically while being unfair in use. For example, sets with equal coverage across groups can give some groups much larger, less useful sets. We aim to identify CP methods and set properties that promote substantive fairness. We build a low-cost evaluation pipeline that uses large language models as decision-makers. It reproduces key patterns from prior human experiments, making it a scalable first-pass tool for evaluating substantive fairness. Across image, text, audio, and tabular data, we compare CP methods and show that clustering labels with similar score patterns reduces set-size gaps across groups and balances utility with substantive fairness. Our findings reveal that similar set sizes across groups correlate strongly with improved substantive fairness, giving practitioners guidance for designing fairer CP systems.