Don't Walk the Line: Boundary Guidance for Filtered Generation

Generative models are increasingly paired with safety classifiers that filter harmful or undesirable outputs. A common strategy is to fine-tune the generator to reduce the probability of being filtered, but this can be suboptimal: it often pushes the model toward producing samples near the classifier’s decision boundary, increasing both false positives and false negatives. We propose Boundary Guidance, a reinforcement learning fine-tuning method that explicitly steers generation away from the classifier’s margin. On a benchmark of jailbreak, ambiguous, and long-context prompts, Boundary Guidance improves the safety while maintaining or improving the utility of outputs, as judged by LLM-as-a-Judge evaluations. Comprehensive ablations across model scales and reward designs demonstrate the robustness of our approach.

Modern AI chatbots are rarely deployed alone. Instead, a separate "safety filter" reviews each response and blocks anything it judges harmful before users see it. The standard way to train the chatbot is to teach it to avoid getting blocked—but this has a subtle problem: it pushes the chatbot to produce some answers that sit right at the edge of what the filter considers acceptable. That edge is exactly where filters make the most mistakes, blocking helpful answers and letting harmful ones slip through. We propose Boundary Guidance, a training method that does the opposite: it rewards the chatbot for producing answers the filter can judge confidently—either clearly safe and useful, or clearly unsafe and easy to reject. Across four AI models, this approach reduces harmful outputs while keeping (or improving) helpfulness, making the overall system more reliable.