VibeSWEBench: Can AI Co-Worker Agents Do Real-World Software Engineering by Vibe Coding?

The growing reliance on autonomous agents for real-world software engineering demands evaluation that reflects actual workflows, where developers give feedback, agents recover from mistakes, and tasks span the full project lifecycle. Existing benchmarks, however, focus largely on single-turn code generation from pristine prompts, ignoring the iterative reality of human–agent collaboration. To address this gap, we introduce **VibeSWEBench**, the first benchmark to evaluate AI co-worker agents on the four core capabilities of real-world development: greenfield construction, feature evolution, specification-driven bug repair, and database operations—and on the multi-turn user-corrective interactions that authentic workflows actually produce. **VibeSWEBench** contains **109** tasks (cumulative weight 230) across two real backend projects, evaluated against comprehensive human-validated test suites with no LLM-as-a-judge component. To capture multi-turn recovery without proxy-developer confounds, we introduce Trajectory Unrolling, a procedure that converts authentic human–agent failure sessions into independent single-turn tasks while preserving human-authored corrective signals. Across four frontier coding agents—Codex, Claude Code, Gemini Code Assist, and Terminus-2—the strongest configuration (Codex with $gpt-5.5$) achieves only **20.87%** overall, with performance dropping from **36.11%** on bug repair to **2.56%** on greenfield tasks, and no successful completions on the hard subset. We release **VibeSWEBench**, its evaluation harness, and all associated artifacts to support future research in autonomous software engineering.