Semantic-level Backdoor Attack against Text-to-Image Diffusion Models

Text-to-image (T2I) diffusion models are widely adopted for their strong generative capabilities, yet remain vulnerable to backdoor attacks. Existing attacks typically rely on fixed textual triggers and single-entity backdoor targets, making them highly susceptible to enumeration-based input defenses and attention-consistency detection. In this work, we propose Semantic-level Backdoor Attack (SemBD), which introduces representation-level triggers based on continuous semantic regions rather than discrete textual patterns. SemBD implants such semantic backdoors by distillation-based editing of the key and value projection matrices in cross-attention layers, enabling semantically equivalent but textually diverse prompts to activate the backdoor. To further enhance stealthiness, SemBD incorporates a semantic regularization to prevent unintended activation under incomplete semantics, as well as multi-entity backdoor targets that avoid highly consistent cross-attention patterns. Extensive experiments demonstrate that SemBD achieves a 100% attack success rate while maintaining strong robustness against state-of-the-art input-level defenses. Our code is available at https://github.com/DPAS-Lab/SemBD/.

Text-to-image models can be secretly changed during training. They may work normally for most prompts, but generate an attacker-chosen image when they see a hidden trigger. Most existing attacks use special words as triggers. This makes them easier to find and block. We show that a trigger can be a meaning, not just a word. Different prompts can describe the same idea in different ways, but still activate the same hidden behavior. We build a method that plants this meaning-based backdoor inside the model while reducing accidental activations. This matters because it shows that backdoors in text-to-image models can hide at the level of meaning, not just words. As a result, defenses that only filter trigger words or look for simple attention patterns may miss serious attacks. Our findings point to the need for safety methods that understand the semantics of prompts and inspect model behavior more deeply.