Constrained Meta Reinforcement Learning with Provable Test-Time Safety

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.

Many real-world decision-making problems, such as those in robotics and autonomous systems, require learning from experience while respecting safety constraints. For example, a robot may need to learn how to move efficiently without colliding with obstacles. A major challenge is to quickly adapt to a new situation while ensuring that safety is never violated during learning. In this work, we study how to use prior experience from related tasks to safely and efficiently learn in a new environment. We design a method that starts from a conservative but safe strategy and gradually improves it by combining it with more rewarding strategies, while maintaining safety at every step. Our approach comes with theoretical guarantees: it ensures safe behavior during learning, converges to an optimal strategy, and uses as few interactions as possible. We also show that these guarantees are essentially the best achievable. Experiments in navigation tasks demonstrate that our method learns faster and achieves higher rewards than existing approaches, while consistently respecting safety constraints.