SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

Search agents connect LLMs to the Internet, enabling them to access broader and more up-to-date information. However, this also introduces a new threat surface: unreliable search results can mislead agents into producing unsafe outputs. Real-world incidents and our two in-the-wild observations show that such failures can occur in practice. To study this threat systematically, we propose SafeSearch, an automated red-teaming framework that is scalable, cost-efficient, and lightweight, enabling sandboxed safety evaluation of search agents. Using this, we generate 300 test cases spanning five risk categories (e.g., misinformation and prompt injection) and evaluate three search agent scaffolds across 17 representative LLMs. Our results reveal substantial vulnerabilities in LLM-based search agents, with the highest ASR reaching 90.5\% for GPT-4.1-mini in a search-workflow setting. Moreover, we find that common defenses, such as reminder prompting, offer limited protection. Overall, SafeSearch provides a practical way to measure and improve the safety of LLM-based search agents.

AI assistants are increasingly able to search the web before answering, which helps them find current information but also creates a new safety problem. Search results can be wrong, misleading, or deliberately written to manipulate the assistant, and the assistant may repeat that information as if it were trustworthy. We study how often this can happen and how serious the failures can be. We introduce SafeSearch, a lightweight testing framework that automatically creates risky search situations in a sandbox, so researchers can evaluate search-based AI systems without exposing real users to harm. SafeSearch generates 300 tests across five kinds of risk, including misinformation and hidden instructions in web pages. We use these tests to evaluate three common ways of building search agents across 17 language models. The results show that many systems are vulnerable: in one setting, the most affected model produced unsafe answers in 90.5% of attacks. Simple defenses, such as reminding the model to be careful, did not reliably solve the problem. Our work gives researchers and developers a practical way to measure these risks and improve search-based AI assistants before they are deployed.