Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors

Existing data poisoning attacks on retrieval-augmented generation (RAG) systems scale poorly because they require costly optimization of poisoned documents for each target phrase. We introduce Eyes-on-Me, a modular attack that decomposes an adversarial document into reusable **Attention Attractors** and **Focus Regions**. Attractors are optimized to direct attention to the Focus Region. Attackers can then insert semantic baits for the retriever or malicious instructions for the generator, adapting to new targets at near zero cost. This is achieved by steering a small subset of attention heads that we empirically identify as strongly correlated with attack success. Across 18 end-to-end RAG settings (3 datasets $\times$ 2 retrievers $\times$ 3 generators), Eyes-on-Me raises average attack success rates from 21.9 to 57.8 (+35.9 points, 2.6$\times$ over prior work). A single optimized attractor transfers to unseen black box retrievers and generators without retraining. Our findings establish a scalable paradigm for RAG data poisoning and show that modular, reusable components pose a practical threat to modern AI systems. They also reveal a strong link between attention concentration and model outputs, informing interpretability research.

Many modern AI systems answer questions by looking up documents in an external digital library. However, this creates a major safety flaw: if a hacker sneaks just one malicious document into that library, they can trick the AI into generating harmful or false answers. Previously, creating these malicious documents was slow and expensive because hackers had to completely rewrite a new document from scratch for every single topic they wanted to target. Eyes-on-Me changes this by breaking the rogue document into two simple, swappable parts: one part acts like a magnet to trick the search engine into picking the document, while the second part contains the specific instructions that influence what the AI actually says. By solving the optimization problem just once, an attacker can swap out topics or malicious commands instantly at zero extra cost. This method improved attack success rates by 2.6 times over prior work and easily fooled major commercial AI systems. This research exposes a critical gap in AI safety, warning developers to better secure AI libraries before these scalable traps are exploited in the wild.