Reflector: Internalizing Step-wise Reflection against Indirect Jailbreaks

While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process. To address these vulnerabilities, we propose Reflector, a principled two-stage framework that internalizes self-reflection within the generation trajectory. Reflector first leverages teacher-guided generation to produce high-quality reflection data for supervised fine-tuning (SFT), establishing structured reflection patterns. It subsequently uses Reinforcement Learning (RL) with outcome-driven and reward-validity supervision to instill robust, autonomous self-reflection capabilities. Empirical results show that Reflector achieves Defense Success Rates (DSR) exceeding 90% against complex indirect attacks while generalizing robustly across diverse threat scenarios. Notably, the framework enhances both task-specific and general utility, yielding a 5.85% gain on GSM8K alongside improved performance on knowledge-intensive benchmarks. By internalizing trajectory-level safety, Reflector overcomes the fundamental limitations of surface alignment without significant computational overhead, offering an efficient and scalable solution for the development of safe and capable LLMs.

As large language models become more capable, they also become more vulnerable to hidden attacks that disguise harmful requests as seemingly harmless instructions. These attacks can bypass existing safety safeguards because current systems often focus only on checking the final response, rather than monitoring the reasoning process that produces it. We introduce REFLECTOR, a training framework that teaches language models to self-monitor while generating responses. Similar to how people pause and reconsider when they realize they may be making a mistake, REFLECTOR enables models to detect risky reasoning patterns, reflect on them, and redirect themselves toward safer responses. To achieve this, we first teach models examples of reflective reasoning and then further train them to recognize when reflection is needed. Experiments show that this approach substantially improves resistance to sophisticated jailbreak attacks, especially indirect attacks designed to evade standard safety defenses, while preserving performance on ordinary tasks. This work highlights a shift in AI safety: instead of only filtering harmful outputs after they are produced, future systems can be trained to identify and correct unsafe reasoning as it happens.