BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

Supervised fine-tuning is the predominant approach for adapting autoregressive vision–language models to downstream tasks. Recent work has shown that this paradigm is highly vulnerable to backdoor attacks, and that existing defenses are ineffective in open-ended generation settings. In response, we propose BYORn, a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible given the corresponding image–text inputs and a pretrained model. BYORn identifies such misaligned responses and dynamically replaces them with alternative responses generated by the model, thereby breaking the correlation between triggers and target outputs. The resulting objective gradient corresponds to the gradient of the empirical estimate of the population risk upper bound over the clean data distribution. Empirically, BYORn consistently improves robustness to backdoor attacks while preserving clean-task performance, establishing a new trade-off frontier between generalization and attack success rate. Finally, we demonstrate that BYORn remains effective against adaptive attacks specifically designed to circumvent the proposed defense.

Modern AI systems that understand both images and text, like the kind that can describe a photo or answer questions about it, are usually customized for specific tasks by training them on example data. But this data can be secretly tampered with. An attacker can plant hidden "triggers" so that the AI behaves normally most of the time, yet produces a chosen malicious response whenever it sees a special pattern in an image. These hidden sabotages, called backdoor attacks, are hard to detect, and existing safeguards largely fail to stop them. We developed BYORn, a training method that defends against this threat. Our key insight is that a tampered answer usually doesn't actually fit the image it's paired with. BYORn spots these mismatched, suspicious answers and, instead of throwing them away, lets the model rewrite them into sensible ones on the fly. This severs the link between the hidden trigger and the attacker's intended response. Across many models and tasks, BYORn makes AI far more resistant to these attacks while keeping its everyday performance intact, and it holds up even against attackers who know exactly how the defense works.