Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Vibe coding is a new software development paradigm in which human engineers prompt a large language model (LLM) agent to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, is the generated code really safe to deploy in production? To investigate this question, we propose SUSVIBES, a benchmark consisting of 186 feature-request software engineering tasks from real-world open-source projects, for which human programmers committed vulnerable implementations. We evaluate 12 widely used coding agentic settings with frontier models on the benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 57% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 11.8% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe coding, particularly in security-sensitive applications. The code and dataset are available at https://github.com/LeiLiLab/susvibes. The leaderboard is at https://leililab.github.io/ susvibes-leaderboard.

AI coding agents can now write entire programs from natural language, which is called "vibe coding." But is the code they produce actually secure? We tested 12 popular AI coding agents on 186 real-world tasks where human developers once made security mistakes. Even the best agent produced functionally correct code 57% of the time, but only 11.8% of solutions were secure. Simple mitigations, like adding security hints to prompts, provided little improvement. Working code is not the same as safe code, and blindly trusting AI coding agents in security-sensitive applications poses serious risks.