Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce **JAILBREAK FOUNDRY (JBF)**, a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) *JBF-LIB* for shared contracts and reusable utilities; (ii) *JBF-FORGE* for the multi-agent paper-to-module translation; and (iii) *JBF-EVAL* for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced$-$reported) attack success rate (ASR) deviation of $+0.26$ percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.

Large language models are tested with “jailbreaks,” carefully designed prompts that try to make a model ignore its safety rules. But new jailbreak methods appear quickly, and many papers test them with different datasets, model settings, and scoring rules, so it is hard to know whether one result can really be compared with another. We built Jailbreak Foundry to make these tests easier to reproduce and compare. It turns jailbreak papers into runnable software modules, checks whether the code matches the paper, and evaluates all attacks using the same benchmark setup. In our experiments, it reproduced 30 published attacks, usually matching the success rates reported in the original papers, and then tested them across 10 language models. The results show that a model may look safe against some attacks while still failing badly against others. This work matters because safety evaluation should not depend on scattered, hard-to-repeat experiments. Jailbreak Foundry can help researchers maintain living benchmarks that track new risks, support authorized safety testing, and stress-test language models more reliably.