Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG

Retrieval-augmented generation (RAG) systems are vulnerable to attacks that inject poisoned passages into the retrieved context, even at low corruption rates. We show that existing attacks are not designed to be stealthy, allowing reliable detection and mitigation. We formalize a distinguishability-based security game to quantify stealth for such attacks. If a few poisoned passages control the response, they must bias the inference process more than the benign ones, inherently compromising stealth. This motivates analyzing intermediate signals of LLMs, such as attention weights, to approximate the influence of different passages on the response. Leveraging attention weights, we introduce the Normalized Passage Attention Score (NPAS) and a lightweight Attention-Variance Filter (AV Filter) that flags anomalous passages. Our method improves robustness, yielding up to ~20% higher accuracy than baseline defenses. We also develop adaptive attacks that attempt to conceal such anomalies, achieving up to 35% success rate and underscoring the challenges of achieving true stealth in poisoning RAG systems.

Modern AI assistants answer questions by retrieving relevant passages from web search or Wikipedia and using them to respond. This lets an attacker plant a misleading poisoned passage, and a single poisoned passage among ten retrieved can be enough to flip the answer. Prior work claimed such attacks are easy to carry out and very hard to defend against. We show they are easier to catch than assumed. Because today's attacks are not designed to be stealthy, the poisoned passage gives itself away: it draws an unusual amount of the model's attention onto the words that carry the wrong answer. We use this telltale fingerprint to build a lightweight filter that removes suspiciously high-attention passages before the answer is generated, and that even protects closed models like GPT-4 via a separate open model. The filter catches the poisoned passage most of the time, cuts attack success sharply, preserves accuracy on normal questions, and beats prior defenses by up to 20%. We also build adaptive attacks that deliberately hide this fingerprint; they sometimes succeed, but only at a steep computational cost, so cheap, effective attacks remain detectable. As retrieval-based AI assistants spread, this makes them substantially harder to manipulate.