Reranker Helps, but Not Enough: Towards Strong Poisoning Attacks Against Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) augments large language models with external knowledge, which in turn exposes their retrieval corpora to data poisoning risks. However, existing poisoning attacks exhibit limited effectiveness against RAG equipped with a reranker to enhance retrieval quality. Remarkably, this defensive capability requires no adversarial training: a reranker fine-tuned solely on benign, in-domain corpora can effectively filter malicious content. Towards realistic RAG red-teaming, we conclude practical prompt design principles that reveal reranker blind spots. Building on these insights, we introduce the Prompt-Perturbation Poisoning Attack ($\mathbf{P}^3 \mathbf{A}$). $\mathbf{P}^3 \mathbf{A}$ first employs rule-based prompt engineering to craft initial poisoned texts. It then injects subtle character-level perturbations into these texts, which promotes their ranking by the reranker while maintaining their adversarial effectiveness. These perturbations introduce only about 1\% textual change, ensuring the poisoned texts remain natural and readable. Extensive experiments show that $\mathbf{P}^3 \mathbf{A}$ achieves strong attack effectiveness and transferability, even when constrained to poisoning a single document. Code is available at https://github.com/YyyxKun/P3A.

Many modern question-answering systems improve their answers by first searching external documents and then using the retrieved information to respond. This makes them more useful and up to date, but it also creates a security risk: an attacker may insert misleading documents into the searchable collection so that the system gives a chosen wrong answer. This paper studies a more realistic version of this threat, where the system also includes an extra step that reorders retrieved documents to keep the most relevant ones. We find that this reordering step can strongly reduce the effectiveness of previous attacks. To better test the safety of such systems, we propose a new attack method that writes convincing misleading documents. It makes very small text changes to help these documents pass the reordering step while remaining readable. Across several datasets, language models, and reordering models, the method is more effective than existing attacks and can remain dangerous even when only one malicious document is inserted. These results show that current defenses are not sufficient and that more careful security testing is needed before such systems are deployed.