The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
Ishan Gonehal ⋅ Hanson Wen ⋅ Bowman Novey
Abstract
As foundation models are deployed for biomedical summarization (multimodal pathology, RNA, and clinical evidence over patient cohorts), evaluation frameworks need to distinguish a model that actually reads the evidence bundle from one whose pretraining priors over the cohort already produce plausible content. The standard supported-claim rate cannot make this distinction: two models can score 87% while differing in clinical safety. We introduce the Hallucination Dependence Index (HDI), a paired metric reporting the fraction of a model's ungrounded hallucination that grounding actually suppresses, computed under bit-for-bit identical prompts with only the evidence bundle substituted between conditions. Pairing HDI with an embedding-overlap probe separates calibrated refusal (low HDI driven by abstention, low overlap) from silent prior-recycling (low HDI driven by reusing baseline content, high overlap). We instantiate HDI in a cross-condition harness on 119 TCGA-LUAD cases anchored to a two-expert consensus ($\kappa = 0.64$; inter-annotator $\kappa = 0.71$), with a fixed external LLM judge (gpt-4.1-mini, in neither factorial row). Across gpt-4o-mini, gpt-5.4-mini, gemini-2.5-flash, and the gemini-3-flash *preview* endpoint, HDI ranges 0.336–0.984 while grounded support compresses to 81.9–93.2%, inverting the safety ranking grounded-only scoring would produce (it prefers gemini-2.5-flash at 93.2% over gemini-3 at 81.9%, yet gemini-3's ungrounded condition produces unsupported claims on 91.5% of cases against 64.8% for gemini-2.5); gpt-5.4-mini's low HDI reflects calibrated refusal, not prior-recycling (3.2% semantic overlap). Pairwise patient-paired bootstrap separates all six model pairs at Holm-corrected $p \leq 0.009$. These structural findings, the ranking inversion and a refusal-driven failure mode, are invisible to grounded-only metrics.
Chat is not available.
Successful Page Load