Hazard Compression: Catastrophic Forgetting in Diffusion-Based Generative Replay under Distribution Shift

Diffusion models trained as generative replay buffers in reinforcement learning are vulnerable to a memorization failure we term hazard compres- sion: as a Lagrangian safety penalty suppresses constraint-violating behavior, hazardous transi- tions vanish from the replay buffer, and the pe- riodically retrained diffusion model catastrophi- cally forgets the constrained region of state-action space. We demonstrate this failure in Prioritized Generative Replay (PGR) (Wang et al., 2025), where the diffusion model’s hazard fidelity (mea- sured by our diagnostic probe, DiffHz) collapses from 13.3% to 0.6% under Lagrangian optimiza- tion. A rare-event memory buffer that preserves hazardous transitions during diffusion retraining resolves this feedback loop, restoring DiffHz to 8.6% and reducing constraint violations by 99.8% on a velocity-constrained locomotion task. On a second task where the Lagrangian multiplier di- verges due to integral windup—a mechanistically distinct failure confirmed by DiffHz remaining high—combined λ-warmup and gradient clipping fully recovers 99.6% of unconstrained reward while reducing cost by 76%. Together, DiffHz and the multiplier trajectory provide a lightweight diagnostic toolkit: low DiffHz signals generative forgetting; diverging λsignals control failure.