Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability

Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ``Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$. Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.

AI cheating is straining our education system; AI "authors" are attempting to supplant human writers, and a scourge of bots is degrading social media. Existing methods to detect AI-generated text are unreliable, expensive, work only on specific models, and/or require large amounts of training data to function. We tackle this by building a more reliable, cheaper, and training-free AI detector that generalizes across models, including ones released after the detector was built. But how? Simple, we find the appendix of the AI models. Let us explain. Humans still have an appendix, we don’t need it, it's a leftover vestigial organ from the process of evolution. During the training of large language models, they too develop heuristics for purposes they no longer need. By identifying one such 'vestigial heuristic', we can use it to separate training-created LLMs from naturally-evolved humans. The result is a fast, broadly effective detector that works even against modern, capable AI models.