CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or scope, and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. To address this gap, we propose CyberGym-E2E, a large-scale and realistic end-to-end cybersecurity benchmark that comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation. CyberGym-E2E is comprehensive and scalable, as we build an automated, agent-enhanced pipeline for transforming open-source vulnerability data into realistic evaluation environments. Currently, the benchmark consists of 920 real-world vulnerabilities across 139 different open-source projects.

AI agents are getting good enough to both find and fix software security vulnerabilities on their own. To know how dangerous or useful these tools really are, we need realistic tests. Existing tests are too small, too artificial, or only cover part of what a real security expert actually does. We built CyberGym-E2E, a collection of 920 real security vulnerabilities drawn from 139 popular open-source software projects. For each one, an AI agent must do what a human security researcher would: locate the bug in a large codebase, craft a small input that verifies the bug, and write a code fix that doesn't break anything else. To build this at the scale we needed, we created a pipeline that uses AI agents themselves to help recreate the needed environments and gather the developer-written tests needed to check that each fix is correct.