Safe Reinforcement Learning with Preference-based Constraint Inference

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy is deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, with great potential in various safety-critical applications.

Training autonomous systems, like self-driving cars or serving robots, to operate safely is highly challenging. While we can train them using human preference feedback, standard AI models often underestimate real-world risks. In safety-critical environments, a minor collision can trigger a catastrophic chain reaction of failures. Standard models overlook these extreme-risk scenarios, leading to dangerously aggressive decisions. To solve this, we developed a new framework called PbCRL. We introduced a dead-zone constraint learning model that mitigates the risk underestimation, forcing the AI to behave safer. We also designed a method to keep the constraint signals clear, and used a two-stage strategy—training on existing data first, then fine-tuning—to avoid constantly bothering human trainers for feedback. We tested PbCRL on robotic tasks, simulated driving, and language model alignment. It successfully trained AIs to complete tasks efficiently while strictly respecting safety boundaries. This work provides a practical, highly secure way to align smart systems with human safety expectations.