Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs

The insatiable demand for web-scale training data has exposed LLMs to a subtle but consequential threat: the absorption of malicious scam content into model weights and its subsequent reproduction during inference. In November 2024, this risk materialized when a developer reportedly lost 2,500 USD after ChatGPT generated an otherwise routine cryptocurrency trading script containing a live phishing URL. To systematically investigate this problem, we introduce Scam2Prompt, an automated auditing framework that crawls known scam websites, infers their functional intent, and synthesizes innocuous developer-style prompts — the kind of legitimate coding requests a programmer might naturally submit — to evaluate whether LLMs reproduce the underlying scam endpoints. Importantly, our approach requires neither jailbreaking nor adversarial prompting; all 1,377 prompts in our benchmark, Innoc2Scam-bench, which is automatically constructed by Scam2Prompt, were human-validated as benign coding tasks. Evaluation of seven production LLMs released in 2025 on Innoc2Scam-bench shows that the vulnerability proves both persistent and severe: malicious code generation rates range from 12.9% to 47.3% across the evaluated models, and no tested model proves immune. State-of-the-art guardrails and RAG-based agents offer only limited protection, underscoring an urgent need for explicit URL validation in LLM-assisted software development pipelines.

AI coding assistants now write a huge share of the world's software, but they learn from the open internet — a place packed with scams. In November 2024, a developer lost 2,500 USD after ChatGPT generated a cryptocurrency trading script that quietly sent his wallet's private key to a phishing website. How widespread is this problem, and which AI assistants are affected? We built Scam2Prompt, an automated detective that hunts for scam links hidden inside popular AI coding tools. Our system writes the kind of innocent coding question any programmer might naturally ask — but on topics that scammers crowd too, like flight booking, coupons, and digital currencies — then checks whether the AI's answer secretly contains a link to a scam site. No tricks or jailbreaking required. We tested seven of the newest AI assistants from major companies including GPT-5 and Gemini 2.5 Pro. Every single one produced code laced with scam URLs, between 13% and 47% of the time. Along the way, our tool uncovered 62 previously unknown live phishing websites that had quietly lurked inside the AI's "memory" for over a year, evading conventional security tools. We released our benchmark publicly so the community can measure progress on fixing this industry-wide blind spot.