The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
Ishan Gonehal ⋅ Hanson Wen ⋅ Bowman Novey
Abstract
We introduce the Hallucination Dependence Index (HDI), a paired metric that reports the fraction of a clinical LLM's ungrounded hallucination that grounding actually suppresses, computed under bit-for-bit identical prompts with only the evidence bundle substituted between conditions. Standard clinical-LLM faithfulness evaluations report a single supported-claim rate, which by construction cannot tell a model that reads the bundle apart from one whose pretraining priors already match the cohort; HDI plus an embedding-overlap probe distinguishes calibrated refusal from silent prior-recycling. We instantiate HDI in a cross-condition harness on 119 TCGA-LUAD cases anchored to a two-expert consensus adjudication ($\kappa = 0.64$; independent inter-annotator $\kappa = 0.71$), with a fixed external LLM judge (gpt-4.1-mini, in neither factorial row) that neutralizes self-preference bias. Across gpt-4o-mini, gpt-5.4-mini, gemini-2.5-flash, and the gemini-3-flash *preview* endpoint, HDI spans 0.336–0.984 while grounded support compresses to 81.9–93.2%, inverting safety rankings; gpt-5.4-mini's low HDI reflects calibrated refusal, not prior-recycling (3.2% semantic overlap). Pairwise patient-paired bootstrap separates all six model pairs at Holm-corrected $p \leq 0.009$. These structural findings are invisible to grounded-only metrics.
Chat is not available.
Successful Page Load