Prompt Injection as Role Confusion

LLMs see the world as a single stream of text, partitioned into roles like <user> or <tool>. We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label. We design role probes to measure how LLMs internally perceive "who is speaking", and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one.

AI assistants follow instructions from different sources: developers set rules, users ask questions, and tools fetch external data like webpages. To stay safe, the AI needs to tell these sources apart. If it can’t, a malicious instruction hidden in a webpage could be mistaken for a legitimate command. Current defenses against such attacks fail routinely against determined adversaries, but the reason has been unclear. We discovered that AI models identify “who is speaking” based on how text sounds, not where it actually comes from. A command hidden in a webpage can hijack an AI agent simply because it sounds like a user's instruction, a failure we call "role confusion". To demonstrate it, we injected fake reasoning into AI prompts. Models misattribute the source, believing this fake reasoning to be their own thoughts, leading to harmful behaviors. We built tools to look inside the model, finding that prompt injections in general succeed when AIs misattribute the source of text. The greater the misattribution, the more likely a prompt injection succeeds. These findings reveal a structural flaw in how AI models process text, and our framework provides tools to measure and address it.