Watershed: A Unified Benchmark for End-to-End Data Provenance Evaluation

Data provenance aims to determine whether and how a data source has influenced a downstream LLM. Despite growing interest in data provenance research, current methods tend to specialize on specific settings and suffer from fragmented evaluation standards. To address this, we introduce WATERSHED, a unified benchmark and toolkit for end-to-end provenance evaluation. WATERSHED structures data provenance into stage-wise tests spanning data preparation, LLM training, black-box auditing, and downstream applications such as membership audit, multi-owner source attribution, and unlearning verification. We evaluate existing provenance methods such as watermarking and membership inference attacks on WATERSHED, across a wide range of datasets, model families and attacks. Our results confirm that methods vary in effectiveness across different stages and tasks. By providing a unified framework and exposing these failure modes, WATERSHED establishes a rigorous basis for evaluating data provenance methods.