MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety

We present MultiBreak, a scalable and diverse multi-turn jailbreak benchmark to evaluate large language model (LLM) safety. Multi-turn jailbreaks mimic natural conversational settings, making them easier to bypass safety-aligned LLM than single-turn jailbreaks. Existing multi-turn benchmarks are limited in size or rely heavily on templates, which restrict their diversity. To address this gap, we unify a wide range of harmful jailbreak intents, and introduce an active learning pipeline for expanding high-quality multi-turn adversarial prompts, where a generator is iteratively fine-tuned to produce stronger attack candidates, guided by uncertainty-based refinement. Our MultiBreak includes 10,389 multi-turn adversarial prompts, spans 2,665 distinct harmful intents, and covers the most diverse set of topics to date. Empirical evaluation shows that our benchmark achieves up to a 54.0% and 34.6% higher attack success rate (ASR) than the second-best dataset on DeepSeek-R1-7B and GPT-4.1-mini, respectively. More importantly, safety evaluations suggest that diverse attack categories uncover fine-grained LLM vulnerabilities, and categories that appear benign under single-turn can exhibit substantially higher adversarial effectiveness in multi-turn scenarios. These findings highlight persistent vulnerabilities of LLMs under realistic adversarial settings and establish MultiBreak as a scalable resource for advancing LLM safety.

Large language models (LLMs) are designed with safety guardrails to refuse harmful requests, but attackers can bypass these protections through multi-turn conversations by gradually steering the LLMs toward dangerous content across multiple exchanges. We built MultiBreak, a large-scale collection of over 10,000 adversarial conversations spanning 2,600+ distinct harmful intents, to rigorously stress-test AI safety. Instead of writing attacks by hand, we trained LLMs to automatically generate and refine them, specifically targeting cases where different models disagree. MultiBreak exposes safety weaknesses that prior benchmarks miss, and reveals that some topics appearing harmless in a single exchange become dangerous when spread across a conversation. We hope it helps researchers build LLMs that stay safe in the realistic, extended conversations people actually have.