Safe In-Context Reinforcement Learning

In-context reinforcement learning (ICRL) is an emerging RL paradigm where an agent, after pretraining, can adapt to out-of-distribution test tasks without any parameter updates, instead relying on an expanding context of interaction history. While ICRL has shown impressive generalization, safety during this adaptation process remains unexplored, limiting its applicability in real-world deployments where test-time behavior is expected to be safe. In this work, we propose SCARED: Safe Contextual Adaptive Reinforcement via Exact-penalty Dual, the first method that promotes safe adaptation of ICRL under the constrained Markov decision process framework. During the parameter-update-free adaptation process, our agent not only maximizes the reward but also keeps the accumulated cost within a user-specified safety budget. We also demonstrate that the agent actively reacts to the safety budget; with a higher safety budget, the agent behaves more aggressively, and with a lower safety budget the agent behaves more conservatively. Across challenging benchmarks, SCARED consistently enables safe and robust in-context adaptation, outperforming existing ICRL and safe meta-RL baselines.

Many AI agents learn by trial and error. A newer approach called in-context reinforcement learning lets an agent keep learning from its recent experience even after it has been deployed, without changing its internal parameters. This flexibility is useful, but it also creates a safety problem: the agent is still experimenting while it is acting, so unsafe exploration can happen during deployment rather than only during training. Existing safe reinforcement learning methods are not designed for this setting, and in our experiments they did not reliably keep the agent safe while it adapted. We introduce SCARED, a new method that trains agents to adapt from experience while also respecting a user-specified safety budget. Across several test environments, SCARED achieved higher task performance while better limiting unsafe behavior than prior baselines. The learned agent is also controllable: giving it a larger cost-to-go budget makes it explore more aggressively and gain more reward, while a smaller budget makes it behave more cautiously.