BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate

Deploying guardrails for custom policies remains challenging, as generic safety models fail to capture task-specific requirements, while prompting LLMs suffers from inconsistent boundary-case performance and high inference costs. Training custom classifiers achieves both accuracy and efficiency, yet demands substantial labeled data that is costly to obtain. We present BARRED (Boundary Alignment Refinement through REflection and Debate), a framework for generating faithful and diverse synthetic training data using only a task description and a small set of unlabeled examples. Our approach decomposes the domain space into dimensions to ensure comprehensive coverage, and employs multi-agent debate to verify label correctness, yielding a high-fidelity training corpus. Experiments across diverse custom policies demonstrate that small language models finetuned on our synthetic data consistently outperform state-of-the-art proprietary LLMs (including reasoning models) and dedicated guardrail models. Ablation studies confirm that both dimension decomposition and debate-based verification are critical for ensuring the diversity and label fidelity required for effective fine-tuning. The BARRED framework eliminates the reliance on extensive human annotation, offering a scalable solution for accurate custom guardrails.

When AI systems are deployed in the real world, in customer service, healthcare, or finance, they need guardrails: automated checks that catch harmful or policy-violating responses. The problem is that every organization has its own rules, and building a custom guardrail traditionally requires collecting and labeling thousands of examples, which is expensive and time-consuming. We propose BARRED, a system that builds custom guardrails from scratch using only a plain-text description of the policy and a small set of unlabeled examples. BARRED works by having AI models brainstorm diverse, tricky test cases that probe the edges of the policy, then uses a panel of AI agents to debate whether each case is labeled correctly — discarding or fixing cases where they disagree. The result is a high-quality training dataset generated entirely by AI, which is then used to train a small, fast, deployable model. In our experiments, these small trained models consistently outperformed much larger AI systems, including state-of-the-art commercial models, at enforcing custom policies accurately. This means organizations can now deploy accurate, efficient guardrails for their specific needs without hiring annotation teams or relying on expensive large models at runtime.