Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation

To deploy large language models (LLMs) in high-stakes application domains that require substantively accurate responses to open-ended prompts, we need reliable, computationally inexpensive methods that assess the trustworthiness of long-form responses generated by LLMs. However, existing approaches often rely on claim-by-claim fact-checking, which is computationally expensive and brittle in long-form responses to open-ended prompts. In this work, we introduce semantic isotropy---the degree of uniformity across normalized text embeddings on the unit sphere---and use it to assess the trustworthiness of long-form responses generated by LLMs. To do so, we generate several long-form responses, embed them, and estimate the level of semantic isotropy of these responses as the angular dispersion of the embeddings on the unit sphere. We find that higher semantic isotropy---that is, greater embedding dispersion---reliably signals lower factual consistency across samples. Our approach requires no labeled data, no fine-tuning, and no hyperparameter selection, and can be used with open- or closed-weight embedding models. Across multiple domains, our method consistently outperforms existing aggregate trust signals in predicting nonfactuality using only a handful of samples, offering a practical, low-cost first-pass signal that complements claim-level verification in real-world LLM workflows.

Large language models are increasingly used to produce long, free-form answers in settings where factual accuracy is essential. This raises a practical question: how can we tell whether such an answer is trustworthy or whether the model is fabricating? The most reliable existing checks are computationally expensive, decomposing each answer into individual claims and verifying every claim against a trusted reference, which can require hundreds of model queries per response. We propose a substantially cheaper alternative. We prompt the model to answer the same question several times and convert each response into a numerical representation of its meaning via a separate embedding model. When the model is on firm ground, these representations align closely; when it is uncertain or fabricating, they tend diverge. The degree of divergence yields a simple score, and we show that greater divergence reliably indicates lower factual accuracy. The approach requires no labelled data, no additional training, and no manual tuning, and is compatible with any standard text-representation model. We additionally introduce a more efficient procedure for grading the factual accuracy of long responses and release a large annotated dataset. Together, these contributions provide a low-cost, scalable signal for flagging unreliable model output before deployment.