Implicit Safety Alignment from Crowd Preferences

Reinforcement Learning from Human Feedback (RLHF) can reveal implicit objectives such as safety considerations that go beyond task completion. In this work, we focus on the common safety criteria embedded in crowd preference datasets, where different users may express distinct preferences or objectives, yet follow similar safety principles. Our aim is to discover shared safety criteria from crowd preferences and then transfer them to downstream RL tasks to regularize agent behavior and enforce safety. We first show that direct reward combination—optimizing a preference-learned reward model together with downstream task rewards—has inherent limitations. Motivated by this, we propose Safe Crowd Preference-based RL, a hierarchical framework that extracts safety-aligned skills from crowd preferences and composes them via a high-level policy to safely solve downstream tasks. Experiments across safe RL environments and a preliminary LLM-style task with diverse user goals and shared safety constraints demonstrate that our approach substantially lowers safety costs without access to explicit safety rewards, while achieving task performance comparable to oracle methods trained with ground-truth safety signals.

Modern AI systems are often trained using feedback from many different people. While users may have different goals or preferences, they frequently share common expectations about safety, such as avoiding harmful or dangerous behavior. In this work, we study how to automatically identify these shared safety principles from human feedback and use them to guide AI systems in new tasks. We show that simply combining safety-related feedback with safety-agnostic task rewards is often unreliable and can lead to poor behavior. To address this, we develop a new framework that first learns reusable safety-related behaviors from crowd feedback and then combines them to solve downstream tasks. This allows the AI system to achieve task goals while also following shared safety expectations, even when explicit safety rules are unavailable. We evaluate our method on several control tasks and a preliminary language-model-style task involving multiple users with different objectives and shared safety requirements. Our results show that the approach can significantly reduce unsafe behavior while maintaining strong task performance.