Active Attacks: Red-teaming LLMs via Adaptive Environments

We address the challenge of automatically generating diverse attack prompts for large language models (LLMs) that elicit harmful behaviors (e.g., insults, sexual content) and are used for safety fine-tuning. While several prior approaches train LLMs with reinforcement learning (RL) to generate such prompts using only a toxicity classifier as a reward, existing diversity-seeking RL methods often collapse to limited modes: once high-reward prompts are found, exploration of new regions is discouraged. Inspired by the active learning paradigm that encourages adaptive exploration, we introduce \textbf{Active Attacks}, a novel RL-based red-teaming algorithm that adapts its attacks as the victim evolves. By periodically safety fine-tuning the victim LLM with collected attack prompts, we naturally induce an \emph{easy-to-hard exploration curriculum}, where the attacker progresses beyond easy modes toward increasingly difficult ones. We observe that this simple plug-and-play module, which seamlessly integrates into existing RL objectives, unexpectedly outperformed prior RL-based methods, improving cross-attack success rates against GFlowNets, the previous state-of-the-art, from 0.07\% to 31.28\% (a relative gain of more than 400×) with only a 6\% increase in computation.

We train LLMs for automated red-teaming, which aims to generate a diverse range of attack prompts that elicit harmful responses from victim LLMs. We observed that prior approaches suffer from a mode collapse issue, as once high-reward prompts are found, exploration of new regions is discouraged. To resolve this issue, we suggest a simple but effective framework called Active Attacks, which adapts its attacks as the victim evolves. Specifically, we periodically reinitialize the attacker and safety fine-tune the victim LLM with the collected attacks so far. We discover that this paradigm naturally emerges in an easy-to-hard curriculum, discovering a diverse range of novel attacks. This RL-based red-teaming algorithm under adaptive environments leads to superior performance without a large computational burden. Our findings can be transferred to detect vulnerabilities of large, closed-source models.