Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

Modern large language models (LLMs) achieve 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching. We introduce clinical reasoning graphs, structured graph representations extracted from free-text LLM diagnostic traces using a domain-grounded ontology with 5 node types and 7 edge types. Applying this extraction pipeline to 750 traces from five LLMs across 50 New England Journal of Medicine Clinicopathological Conference cases and three prompt conditions, we test whether LLM diagnostic traces show stable structured reasoning patterns, i.e., "illness scripts", for clinically similar cases, determined by higher graph similarity among clinically similar cases than among clinically dissimilar cases. Across 15 model–condition comparisons, within-cluster and between-cluster graph similarity are nearly equal, with no comparison surviving multiple-testing correction. Graph similarity was also similar for pairs of models that were both correct versus both incorrect, suggesting that graph structure captures a dimension not reflected in diagnosis accuracy. Structured reflection prompting increased explicit discriminating-feature analysis within traces, but did not increase cross-case consistency. We release the ontology, extraction pipeline, validation protocol, and 750 extracted graphs as resources for structured evaluation of LLM clinical reasoning.