GAUSS: Graph-Assisted Uncertainty Quantification using Structure and Semantics for Long-Form Generation in LLMs

In critical domains like clinical reporting, legal analysis, and policy drafting, large language models (LLMs) are increasingly expected to produce extended, fact‑rich narratives rather than isolated sentences. Reliable uncertainty quantification in such long‑form outputs is crucial. Existing techniques either assign a single confidence score to an entire paragraph or evaluate factual consistency by comparing extracted atomic facts across multiple generations. Some recent approaches represent fact–paragraph relationships using bipartite entailment graphs and derive uncertainty from fact centrality. However, these methods ignore the explicit dependencies among facts within a paragraph and the structural and semantic variation across multiple LLM outputs for the same prompt, missing a key source of uncertainty specific to long‑form generation. We propose GAUSS (Graph‑Assisted Uncertainty Quantification using Structure and Semantics), a principled framework that models each generated paragraph as a semantic graph of atomic facts and their relations. We posit that uncertainty arises from structural and semantic discrepancies among these graphs across different samples. GAUSS quantifies uncertainty as the expected alignment cost between the semantic graph of an anchor paragraph and those of alternative generations. By capturing both semantic content and structural coherence, GAUSS offers a more interpretable and theoretically grounded measure of uncertainty than coarse, sentence‑level scores.

Large language models are increasingly used to write detailed answers in areas such as healthcare, law, and public policy, but they can produce convincing text that is partly incorrect. This paper addresses a basic question: how can we tell when a long answer from an AI system may be unreliable? We introduce GAUSS, a method that asks an AI system for several answers to the same question and compares both the facts they contain and how those facts fit together. When the answers disagree strongly in meaning or organization, GAUSS reports higher uncertainty, signaling that the output may need closer checking. Across several evaluation datasets and language models, GAUSS was more effective than existing approaches at identifying answers that were likely to contain factual errors. This work could help make long AI-generated responses safer to use by providing a clearer warning when users should verify the content before relying on it.