Timezone: »

Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments
Yixuan Wang · Simon Zhan · Ruochen Jiao · Zhilu Wang · Wanxin Jin · Zhuoran Yang · Zhaoran Wang · Chao Huang · Qi Zhu

Wed Jul 26 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #232

It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions. Many popular safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in a cost function and try to constrain the expectation of cumulative cost under a threshold. However, it is often difficult to effectively capture and enforce hard reachability-based safety constraints indirectly with such constraints on safety violation cost. In this work, we leverage the notion of barrier function to explicitly encode the hard safety chance constraints, and given that the environment is unknown, relax them to our design of generative-model-based soft barrier functions. Based on such soft barriers, we propose a novel safe RL approach with bi-level optimization that can jointly learn the unknown environment and optimize the control policy, while effectively avoiding the unsafe region with safety probability optimization. Experiments on a set of examples demonstrate that our approach can effectively enforce hard safety chance constraints and significantly outperform CMDP-based baseline methods in system safe rates measured via simulations.

Author Information

Yixuan Wang (Northwestern University)
Simon Zhan (University of California, Berkeley)
Ruochen Jiao (Northwestern University, Northwestern University)
Zhilu Wang (Northwestern University)
Wanxin Jin (University of Pennsylvania)
Zhuoran Yang (Yale University)
Zhaoran Wang (Northwestern University)
Chao Huang (University of Liverpool)
Qi Zhu (Northwestern University)

More from the Same Authors