Proactive Defense Benchmark against Deepfake Generation

Despite the proliferation of proactive defenses against deepfakes, the lack of a unified evaluation protocol precludes fair comparison and masks critical vulnerabilities. To bridge this gap, we present the first comprehensive benchmark that systematically assesses disruption, robustness, and transferability encompassing pixel, perceptual, and identity metrics. Our extensive analysis reveals that fidelity and identity metrics capture orthogonal performance axes, often leading to conflicting interpretations when relied upon individually. Furthermore, we identify a fundamental trade-off where peak white-box performance signals overfitting, and we introduce a calibrated evaluation to correct generator-induced identity bias. By exposing these blind spots, we establish a rigorous standard to guide the development of genuinely generalizable protections. Project page is available at: https://proactivedefensebenchmark.github.io/

Can we protect our online photos from being turned into deepfakes before they are even shared? One promising idea is to apply an invisible filter to a photo so that any AI model attempting to manipulate it produces a distorted, unusable result. Many such "proactive defenses" have been proposed, but each study evaluates them differently, leaving it unclear which methods actually work in practice. We built the first comprehensive benchmark to compare these defenses on equal footing. Rather than relying on a single number, we evaluated each method from multiple angles: how much the deepfake output was visually corrupted, how much the person's identity was changed, and whether the defense still held up after everyday image processing such as JPEG compression or against deepfake models it had never encountered before. Our analysis reveals several surprising blind spots in how proactive defenses are currently evaluated. Among the most striking: measures of visual corruption and measures of identity change often disagree, meaning a defense that looks strong by one standard can look weak by another. Equally telling, defenses that achieve the highest scores when the attacker's model is fully known tend to overfit to that ideal setting and fail against new, unseen deepfake models, much like a student who aces practice problems but struggles on the real exam. We also introduce a corrected metric that removes a hidden bias coming from the deepfake generator itself. By exposing these blind spots, our benchmark establishes a rigorous standard for building proactive defenses that can genuinely protect people in the real world.