Hazard Compression: Catastrophic Forgetting in Diffusion-Based Generative Replay for Safe Reinforcement Learning

Generative replay accelerates online RL by train- ing a generative model on accumulated experi- ence and densifying that experience with synthetic transitions—an emerging paradigm for sample- efficient online learning. We show that this paradigm has a critical safety failure mode un- der constrained optimization: when a Lagrangian penalty suppresses constraint-violating behav- ior, hazardous transitions vanish from the replay buffer, and the periodically retrained diffusion model catastrophically forgets the constrained re- gion of state-action space—a phenomenon we term hazard compression. We demonstrate this failure in Prioritized Generative Replay (Wang et al., 2025), where the diffusion model’s haz- ard fidelity (measured by our diagnostic probe, DiffHz) collapses from 13.3% to 0.6% under Lagrangian optimization. A rare-event memory buffer that preserves hazardous transitions during diffusion retraining resolves this feedback loop, restoring DiffHz to 8.6% and reducing constraint violations by 99.8% on a velocity-constrained locomotion task. On a second task where the Lagrangian multiplier diverges due to integral windup—a mechanistically distinct failure con- firmed by DiffHz remaining high—combined λ-warmup and gradient clipping fully recovers 99.6% of unconstrained reward while reducing cost by 76%. Together, DiffHz and the multiplier trajectory provide a lightweight diagnostic toolkit for diffusion-based generative replay: low DiffHz signals generative forgetting; diverging λsignals control failure.