CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95\% solution correctness and 96\% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2\% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3\% to 35.8\% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5\% to 31.3\%). We open-source all code, data, and models.

Public software vulnerability reports often contain only brief descriptions and links, but not the complete environments needed to safely reproduce, test, and fix the problems. This makes it expensive for experts to build realistic security tasks for evaluating AI coding agents. We introduce CVE-Factory, an automated system that turns public vulnerability reports into executable security tasks with environments, tests, and reference fixes. CVE-Factory uses multiple specialized agents and step-by-step verification to make this process scalable and reliable. Using it, we build LiveCVEBench, a continuously updated benchmark of recent real-world vulnerabilities, and generate large-scale training data for security-oriented coding agents. Our experiments show that these generated tasks can significantly improve an open-source coding model and help evaluate whether AI agents can handle emerging software security problems.