Extracting alignment data in open models

In this work, we show that it is possible to extract significant amounts of alignment training data from a post-trained model -- useful to steer the model to improve certain capabilities such as long-context reasoning, safety, instruction following, and maths. While the majority of related work on memorisation has focused on measuring success of training data extraction through string matching, we argue that embedding models are better suited for our specific goals. Distances measured through a high quality embedding model can identify semantic similarities between strings that a different metric such as edit distance will struggle to capture. In fact, in our investigation, approximate string matching would have severely undercounted (by a conservative estimate of $10\times$) the amount of data that can be extracted due to trivial artifacts that deflate the metric. Interestingly, we find that models readily regurgitate training data that was used in post-training phases such as SFT or RL. We show that this data can be then used to train a base model, recovering a meaningful amount of the original performance. We believe our work exposes a possibly overlooked risk towards extracting alignment data. Finally, our work opens up an interesting discussion on the downstream effects of distillation practices: since models seem to be regurgitating aspects of their training set, distillation can therefore be thought of as indirectly training on the model's original dataset.

Modern AI assistants learn their advanced reasoning and safe behaviour through highly guarded, custom datasets. This "alignment data" is a developer's secret recipe and a massive competitive advantage. But what if the AI inadvertently leaks this secret recipe to the public or competitors? We discovered that by starting a prompt with specific, hidden formatting tags—the exact same tags developers use behind the scenes to structure AI conversations—we can trick openly available AI models into reciting their original training examples. To measure the true scale of this leakage, we moved away from traditional tests that only look for exact word-for-word matches. Instead, we used an AI tool that evaluates the meaning and structure of the text, allowing us to catch instances of "semantic memorisation" where the model changes a few trivial words but leaks the core data. Our method revealed that models leak at least ten times more training data than previously believed. In fact, the leaked data is of such high quality that we successfully used it to train a brand-new AI model. This proves that releasing an open AI model risks giving away the expensive, proprietary data used to build it, threatening intellectual property and exposing a major blind spot in how the industry measures data leakage.