When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models

Large Language Models (LLMs) have been augmented with web search to overcome the limitations of the static knowledge boundary by accessing up-to-date information from the open Internet. While this integration enhances model capability, it also introduces a distinct safety threat surface: the retrieval and citation process has the potential risk of exposing users to harmful or low-credibility web content. Existing red-teaming methods are largely designed for standalone LLMs as they primarily focus on unsafe generation, ignoring risks emerging from the complex search workflow. To address this gap, we propose CREST-Search, a pioneering red-teaming framework for LLMs with web search. The cornerstone of CREST-Search is three novel attack strategies that generate seemingly benign search queries yet induce unsafe citations. It also employs an iterative in-context refinement mechanism to strengthen adversarial effectiveness under black-box constraints. In addition, we construct a search-specific harmful dataset, WebSearch-Harm, which enables fine-tuning a specialized red-teaming model to improve query quality. Our experiments demonstrate that CREST-Search can effectively bypass safety filters and systematically expose vulnerabilities in web search-based LLM systems, underscoring the necessity of the development of robust search models.

Modern AI systems increasingly use web search to answer questions with up-to-date information from the Internet. While this improves their usefulness, it also creates new safety risks: these systems may retrieve and cite harmful online content. Existing safety testing methods mainly focus on whether AI models generate unsafe text directly, often overlooking risks introduced by the web search process. In this work, we introduce CREST-Search, a framework designed to systematically test the safety of AI systems augmented with web search. Our method creates search queries that appear harmless but can still guide AI systems to cite unsafe or unreliable web content. We also develop a new dataset focused on harmful web search scenarios and use it to improve the quality of these safety tests. Our experiments show that current search-enabled AI systems can still be manipulated into retrieving harmful information, even when the systems’ responses themselves appear safe. These findings highlight the importance of developing more reliable and secure web-connected AI systems as they become more widely used.