Configurable Reward Model for Balanced Safety Alignment

Aligning large language models (LLMs) to heterogeneous and rapidly evolving safety requirements remains a critical challenge. Existing instruction-tuned LLMs and standalone safety classifiers often fail to generalize to new safety configurations, motivating the need for Reward Models (RMs) that are explicitly configurable to changing specifications. We introduce the Configurable Safety Reward Model (CSRM), which is jointly optimized for calibrated safety compliance and reward modeling. Our approach is supported by configuration-targeted data augmentation that enforces instruction adherence while preserving relative severity structure. The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations. CSRM achieves state-of-the-art performance on recent configurable safety benchmarks, including CoSApien (94.6\% F1) and DynaBench (75.8\% F1), without requiring additional human annotation. When used for downstream safety alignment, CSRM yields LLMs with a significantly improved helpfulness–safety tradeoff compared to existing baselines.

Large language models are increasingly used in settings where safety rules matter, but those rules are not fixed: different applications, organizations, and communities may define harmful or acceptable behavior differently. Today’s safety systems often struggle when the rules change, because they are usually trained for one fixed policy or a small set of known cases. We introduce the Configurable Safety Reward Model, or CSRM, a system that can judge model responses according to different safety instructions. Instead of treating safety as a single yes-or-no decision, CSRM learns to account for fine-grained differences in policy requirements, severity, and conversational context. We also use targeted data augmentation so the model can better follow new safety configurations without requiring extra human-labeled data. Across recent safety benchmarks, CSRM performs better than prior methods, including on settings it has not seen during training. When used to train language models, it improves the balance between being helpful and staying safe, making it easier to adapt AI systems to evolving safety needs.