Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

Fine-tuning APIs offered by major AI providers create new attack surfaces where adversaries can bypass safety measures through targeted fine-tuning. We introduce Trojan-Speak, an adversarial fine-tuning method that bypasses Anthropic's Constitutional Classifiers. Our approach uses curriculum learning combined with GRPO-based hybrid reinforcement learning to teach models a communication protocol that evades LLM-based content classification. Crucially, while prior adversarial fine-tuning approaches report more than 25\% capability degradation on reasoning benchmarks, Trojan-Speak incurs less than 5\% degradation while achieving 99+\% classifier evasion for models with 14B+ parameters. We demonstrate that fine-tuned models can provide detailed responses to expert-level CBRN (Chemical, Biological, Radiological, and Nuclear) queries from Anthropic's Constitutional Classifiers bug-bounty program. Our findings reveal that LLM-based content classifiers alone are insufficient for preventing dangerous information disclosure when adversaries have fine-tuning access, and we show that activation-level probes can substantially improve robustness to such attacks.

AI companies let customers retrain models through customization services, relying on safety filters that scan the training data and conversations to block harmful content. In earlier attacks, retraining a model to evade these filters cost it over a quarter of its reasoning ability, a "jailbreak tax" that limited the danger. Working with Anthropic to find weaknesses before attackers do, we show this tax can be avoided. Using such a service, we taught a model to write in a simple secret code: a handful of common letters swapped, the text dressed up as technical logs. It then gave detailed answers to expert-level chemical and biological weapons questions while keeping roughly 95 percent of its scores on standard reasoning tests. More than 99 percent of the coded answers slipped past Constitutional Classifiers, Anthropic's deployed safety filter, which withstood over 3,000 hours of professional attack testing. But detectors reading the model's internal activity, not its words, caught the hidden content almost perfectly, even when we tried to train the model to fool them. Today's most capable models do not yet offer this kind of retraining; we argue for adding such internal checks alongside text filters before that access expands.