Black-Box Detection of LLM-Generated Text Using Generalized Jensen Shannon Divergence

We study black-box detection of machine-generated text under practical constraints: the scoring model (proxy LM) may mismatch the unknown source model, and per-input contrastive generation is costly. We propose SurpMark, a reference-based detector that summarizes a passage by the dynamics of its token surprisals. SurpMark discretizes surprisals into interpretable states, estimates a state-transition matrix for the test text, and scores it via a generalized Jensen–Shannon (GJS) gap between the test transitions and two fixed references (human vs. machine) built once from existing corpora. Theoretically, we derive design guidance for how the discretization bins should scale with data and provide a principled justification for our test statistic. Empirically, across multiple datasets, source models, and scenarios, SurpMark consistently matches or surpasses baselines, demonstrating strong robustness across domains and generators; our experiments on hyperparameter sensitivity exhibit trends that our theoretical results help to explain.

As AI-generated text becomes increasingly common, it is important to develop reliable ways to distinguish it from human-written text. This paper studies machine-generated text detection by looking at how predictable or surprising each token is under a language model. We propose SurpMark, a method that compares the patterns of these token-level surprise signals between human-written and machine-generated text. Across multiple datasets, generators, and robustness settings, SurpMark performs better than existing detection methods.