MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations

Large Language Models (LLMs) have inherent limitations of faithfulness and factuality, commonly referred to as hallucinations. Several benchmarks have been developed that provide a test bed for factuality evaluation within the context of English-centric datasets, while relying on supplementary informative context like web links or text passages but ignoring the available structured factual resources. To this end, Knowledge Graphs (KGs) have been identified as a useful aid for hallucination mitigation, as they provide a structured way to represent the facts about entities and their relations with minimal linguistic overhead. We bridge the lack of KG paths and multilinguality for factual language modeling within the existing hallucination evaluation benchmarks and propose a KG-based multilingual, multihop benchmark called MultiHal framed for generative text evaluation. As part of our data collection pipeline, we mined 140k KG-paths from open-domain KGs, from which we pruned noisy KG-paths, curating a high-quality subset of 25.9k. Our baseline evaluation shows an absolute scale improvement by approximately 0.12 to 0.36 points for the semantic similarity score, 0.16 to 0.36 for NLI entailment and 0.29 to 0.42 for hallucination detection in KG-RAG over vanilla QA across multiple languages and multiple models, demonstrating the potential of KG integration. We anticipate MultiHal will foster future research towards several graph-based hallucination mitigation and fact-checking tasks.

Large language models have a fundamental flaw of "hallucinating" - confidently producing factually inconsistent text that looks convincing. While tools exist to test these errors, they mostly focus on English and rely on unstructured text in forms of text documents or web pages rather than clean, organized databases. To bridge this gap, we created MultiHal, a new multi-language benchmark that builds upon previously established test benchmarks and accompanies them with data from a structured source called Wikidata Knowledge Graph. We harvested and verified over 25,000 high-quality factual connections from English and performed translation to Spanish, German, Italian, Portuguese, and French. When we fed these structured facts to AI models, their factual accuracy raises significantly while hallucinations plummeted across the board, demonstrating the usefulness of the resource. MultiHal gives the scientific community a tool to test AI for its factual consistency, specifically using structured data, making future systems more trustworthy.