From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defense

Deep Neural Networks (DNNs) remain fundamentally vulnerable to backdoor attacks. Traditional data-free defenses largely operate under the paradigm of internal diagnosis methods like model repairing or input robustness, yet these approaches are often fragile under advanced attacks as they remain entangled with the victim model’s corrupted parameters. We propose a paradigm shift to data-free External Semantic Auditing, using universal Vision-Language Models (VLMs) as independent auditors to decouple defense from the compromised model. We introduce PRISM (Prototype Refinement & Inspection via Statistical Monitoring), which transforms generic VLMs into domain-adaptive gatekeepers purely via online test-time adaptation. PRISM bridges the domain gap through a Hybrid VLM Teacher that refines prototypes from the test stream and an Adaptive Router that calibrates thresholds via statistical monitoring. Evaluation across 17 datasets and 11 attack types confirms PRISM achieves state-of-the-art performance (suppressing Attack Success Rate to < 1% on CIFAR-10), proving that robust defense is achievable without touching the model weights or accessing a single training sample.

AI models can be secretly sabotaged. A malicious actor can embed a hidden "backdoor" during training, causing the model to misclassify any image stamped with a specific trigger (such as a small patch or subtle texture), while behaving perfectly normally otherwise. Detecting this sabotage is extremely difficult once the model is deployed, especially when the defender has no access to the original training data. We built PRISM, a real-time security system that watches over a potentially compromised AI model without needing any training data. The key idea is to bring in an independent, trusted witness: a large Vision-Language Model (like CLIP) that was trained on billions of internet images and has broad knowledge of the visual world. PRISM compares the suspicious model's predictions against the independent witness on every incoming image. When the two disagree in a statistically unusual way, a telltale sign of a backdoor trigger, PRISM blocks the prediction before it reaches the user. Because the witness is trained completely separately from the model being audited, a saboteur cannot fool both simultaneously without an unrealistic level of coordination. PRISM requires no retraining, no labeled data, and adds only milliseconds of latency, making it practical for real-world AI services where trust cannot be taken for granted.