Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

Large language models frequently exhibit hallucinations: fluent and confident outputs that are factually incorrect or unsupported by the input context. While recent hallucination detection methods have explored various features derived from attention maps, the underlying mechanisms they exploit remain poorly understood. In this work, we propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are deeply entangled with attention sinks - tokens that accumulate disproportionate attention mass during generation - indicating a transition from distributed, input-grounded attention to compressed, prior-dominated computation. Importantly, although sink scores are computed solely from attention maps, we find that the classifier preferentially relies on sinks whose associated value vectors have large norms. Moreover, we show that previous methods implicitly depend on attention sinks by establishing their mathematical relationship to sink scores. Our findings yield a novel hallucination detection method grounded in theory that produces state-of-the-art results across popular datasets and LLMs.

Although Large Language Models are increasingly ubiquitous, they frequently generate falsehoods—a flaw called "hallucination" that limits their use in high-stakes fields like medicine, law, and finance. External verification is often tedious and resource-intensive. However, a compelling alternative exists: what if we could catch these falsehoods by looking inside the model, tapping into its own internal signals much like we do when a human is unsure? We discovered that right before a model hallucinates, its "attention" undergoes a distinct disruption, getting trapped on specific words instead of processing the sentence naturally. By tracking this breakdown using a metric called a "sink score," we established a direct link between these internal focus traps and the generation of falsehoods. We leverage this phenomenon in SinkProbe, a tool that monitors these hidden warning signals to catch hallucinations at their source. By providing a highly efficient tool to intercept hallucinations before they reach the user, SinkProbe offers a practical pathway toward deploying reliable AI. Ultimately, this discovery deepens our understanding of the inner workings of LLMs.