DIS-CO: Discovering Copyrighted Content in VLMs Training Data

How can we verify whether copyrighted content was used to train a large vision-language model (VLM) without direct access to its training data? Motivated by the hypothesis that a VLM is able to recognize images from its training corpus, we propose DIS-CO, a novel approach to infer the inclusion of copyrighted content during the model's development. By repeatedly querying a VLM with specific frames from targeted copyrighted material, DIS-CO extracts the content's identity through free-form text completions. To assess its effectiveness, we introduce MovieTection, a benchmark comprising 14,000 frames paired with detailed captions, drawn from films released both before and after a model’s training cutoff. Our results show that DIS-CO significantly improves detection performance, nearly doubling the average AUC of the best prior method on models with logits available. Our findings also highlight a broader concern: all tested models appear to have been exposed to some extent to copyrighted content. We provide the code in the supplementary materials.

Large Vision-Language Models learn by training on huge collections of images and text. However, there is growing concern that these collections might include copyrighted material, raising important legal and ethical questions.Motivated by the idea that models can recognize images they have seen before, we developed DIS-CO: a new approach for detecting whether a model has memorized specific visual content, with a special focus on copyrighted material.DIS-CO works by repeatedly prompting the model to identify the source of carefully selected images: for example, asking, “Which movie is this frame from?”, and requiring the model to generate its answer freely, instead of picking from multiple-choice options. This free-form format is crucial: it makes it extremely unlikely for the model to guess the correct answer by chance. If the model can reliably provide the correct titles for challenging frames, it offers strong evidence that it encountered this content during training.To rigorously evaluate DIS-CO, we created MovieTection, a benchmark featuring 14,000 frames from 100 movies, and our results are clear: all tested models showed signs of exposure to copyrighted content. With DIS-CO, even when companies do not disclose their training data, it becomes possible to reveal whether specific copyrighted material was used. As such, we believe this offers a new path toward greater transparency and accountability in AI development.