Fingerprinting Pre-trained Encoders under Arbitrary Downstream Fine-Tuning via Adversarial Shifting

In the pre-training-fine-tuning paradigm, pre-trained encoders have become high-value intellectual property (IP) due to their immense training costs, necessitating robust protection. Existing fingerprinting or watermarking methods typically rely on pre-defined samples and labels, or require intrusive modifications to the training process. However, downstream fine-tuning can significantly alter an encoder's representation and label space, thereby destroying the label consistency of existing methods and rendering them ineffective. Consequently, it is both challenging and urgent to provide a downstream-agnostic, black-box ownership verification mechanism for pre-trained encoders. To address this, we propose a downstream-agnostic, label-only fingerprinting method that leverages Adversarial Shifting to construct stable fingerprint clusters in the encoder’s latent space. By exploiting the inherent output consistency of these clusters, our method remains effective regardless of the specific downstream task or label mapping. Extensive experiments demonstrate that our method maintains superior robustness and stealthiness across various downstream tasks and category scales, providing a practical and reliable IP protection scheme for high-value pre-trained encoders.

Training AI foundation models is costly, making them valuable intellectual property prone to theft. Thieves often steal and retrain core models for new tasks, breaking traditional "fingerprints" because the model's final outputs change fundamentally. To address this, we introduce a resilient fingerprinting method that survives these modifications. Instead of relying on fragile output labels, we hide our fingerprint deep within the model's internal understanding. We subtly alter a group of input images so the AI views them as identical, forming a hidden "data island". Because these samples are grouped so closely, any stolen model will always categorize them together, regardless of its new task. By simply checking if a suspect model gives these samples the same answer, we can reliably prove ownership and protect AI investments.