Rethinking Pretraining Data Detection for LLMs: From Local to Global

The advancements of Large Language Models (LLMs) are primarily attributed to massive pretraining data, which also introduces risks like privacy leakage and data contamination. Therefore, it is crucial to determine whether an LLM has been trained on a given target text. Existing detection methods primarily rely on local statistics of isolated tokens (e.g., those with the lowest probabilities), neglecting the probability dynamics during the token generation process. In this paper, we shift the detection paradigm from a local token to a global sequence perspective, grounded in the core intuition that memorized sequences exhibit volatility patterns distinct from those generated via inference. We propose Adaptive Entropic Convolutional Analysis (AECA), a framework that conceptualizes the probability sequence as a dynamic signal, integrating calibration with convolutional filtering to effectively capture memorization signals. Extensive experiments demonstrate that AECA surpasses previous methods by up to 1.5\% in average AUC on the WikiMIA benchmark, with its advantage being particularly pronounced in long-text scenarios.

Large language models are trained on enormous collections of text, but the exact contents of these collections are often unknown. This creates an important question: has a model already seen a particular piece of text during training? Answering this question matters for protecting privacy, checking possible copyright issues, and making sure that AI benchmarks are fair. Existing methods usually look at a few individual words that the model finds surprising. However, this can miss a broader pattern: when a model is recalling text it has memorized, its confidence may change differently from when it is generating text based on general understanding. In this work, we propose a new method that looks at the model’s confidence across the whole text, rather than focusing only on isolated words. Our method treats these confidence changes like a signal and searches for patterns that suggest memorization. Experiments show that our approach is more accurate than previous methods, especially for longer texts. This work can help researchers and developers audit LLMs more transparently and responsibly.