Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed

Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.

Safe reinforcement learning algorithms can learn to make safe decisions over time, but in safety-critical settings it is not enough for them to be safe only under the exact conditions seen during training. We study this problem in diabetes management, where treatment-support decisions must remain reliable for patients with different glucose and insulin dynamics. We build a unified diabetes simulator and use it to evaluate eight safe reinforcement learning algorithms across different diabetes types and age groups. We reveal a safety generalization gap: algorithms that satisfy safety constraints on training patients can still make unsafe decisions on unseen patients. To address this gap, we introduce a predictive shielding mechanism that intervenes before an action is taken. The shield uses recent patient history to adapt glucose forecasts, predicts how candidate actions may affect future glucose levels, and filters actions that could lead to unsafe outcomes. This allows the controller to account for patient-level variation without retraining the policy online. Across 72 simulated clinical settings, this predictive shielding improves Time-in-Range, reduces clinical risk and glucose variability, and works across multiple safe RL algorithms. Our simulator and benchmark provide a reusable platform for studying safety under patient-level distribution shift.