Training with Honeypots: Reshaping How LLMs Fail Under Adversarial Attacks

Automated red-teaming of Large Language Models (LLMs) commonly relies on attack success rates (ASR) as a proxy for real-world harm, implicitly assuming that judge-detected violations correspond to actionable risk. In practice, safety judges are imperfect, and outputs that satisfy automated criteria for harm can vary widely in their operational usefulness. In this work, we investigate whether model failure modes can be reshaped so that, when defenses fail, they preferentially produce reduced-actionability outputs rather than highly actionable harm. Inspired by honeypots in computer security, we construct responses that are frequently flagged as harmful by automated judges yet provide limited real-world operational value, and treat them as hard negatives in the safety training pipeline. Our findings show that shaping how models fail under attack can improve overall safety by reducing both the real-world impact and the frequency of harmful failures, and serves as a practical complement to ASR-based evaluations.

Large language models can be tricked into giving harmful answers, even when they are trained to refuse unsafe requests. Many safety evaluations simply focus on whether an attack succeeded, but not all failures are equally dangerous: a vague or unusable answer is much less harmful than a detailed, practical one. In this paper, we study whether models can be trained to fail in safer ways when attacks get through. We create "honeypot" answers that look harmful to automated safety checkers but are not useful for carrying out real-world harm, and use them during training to steer models away from more actionable harmful responses. Across several models, defenses, attacks, and safety evaluators, this training reduces on average both the number of successful attacks and the usefulness of the remaining harmful-looking outputs, while preserving performance on normal tasks.