Training Language Model Agents to Find Vulnerabilities with CTF-Dojo

Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through verified feedback loops. Yet, scalable and generalizable execution-grounded environments remain scarce, limiting progress in training more capable ML agents. We introduce CTF-Dojo, the first large-scale executable runtime tailored for training LLMs with verifiable feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style challenges containerized in Docker with guaranteed reproducibility. To enable rapid scaling without manual intervention, we develop CTF-Forge, an automated pipeline that transforms publicly available artifacts into ready-to-use execution environments in minutes, eliminating weeks of expert configuration traditionally required. We trained LLM-based agents on just 486 high-quality, execution-verified trajectories from CTF-Dojo, achieving up to 11.6% absolute gains over strong baselines across three competitive benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1, establishing a new open-weight state-of-the-art that rivals frontier models like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a benchmark for executable-agent learning, CTF-Dojo demonstrates that execution-grounded training signals are not only effective but pivotal in advancing high-performance ML agents without dependence on costly proprietary systems.

Modern cybersecurity increasingly depends on AI systems that can analyze software, identify vulnerabilities, and interact with complex computer environments. However, training capable cybersecurity agents remains difficult because there are very few large-scale environments where models can safely practice realistic security tasks and receive reliable feedback on whether they succeed or fail. Our work introduces CTF-Dojo, a large collection of reproducible Capture-The-Flag (CTF) cybersecurity challenges packaged into isolated Docker environments. These challenges allow AI agents to practice tasks such as vulnerability discovery and exploitation in realistic but controlled settings. To make building these environments scalable, we also develop Forge, an automated pipeline that converts public CTF artifacts into executable runtime environments within minutes. Using only a few hundred verified training trajectories collected from CTF-Dojo, we train open-source AI models that substantially improve performance on established cybersecurity benchmarks and approach the capability of much larger proprietary systems. We additionally study practical techniques for improving cybersecurity agents, including using public writeups as hints and varying runtime environments during training. Our findings suggest that execution-grounded training environments can provide an effective and scalable foundation for future cybersecurity AI research.