SMD: Multi-view Safety-Critical Driving Video Generation in the Real-world Domain

Safety-critical scenarios are essential for evaluating autonomous driving (AD) systems, yet they are rare in practice. Existing generators produce trajectories, simulations, or single-view videos—but they don’t meet what modern AD systems actually consume: realistic multi-view video. We present SMD, the first framework for generating multi-view safety-critical driving videos in the real-world domain. SMD couples a safety-critical trajectory engine with a diffusion-based multi-view video generator through three design choices. First, we pick the right adversary: a GRPO-fine-tuned vision-language model (VLM) that understands multi-camera context and selects vehicles most likely to induce hazards. Second, we generate the right motion: a two-stage trajectory process that (i) produces collisions, then (ii) transforms them into natural evasion trajectories—preserving risk while staying within what current video generators can faithfully render. Third, we synthesize the right data: a diffusion model that turns these trajectories into multi-view videos suitable for end-to-end planners. Videos generated by SMD substantially increase collision rates when stress testing multiple end-to-end planners, and reduce collision rates when incorporated into training, improving planner robustness and safety. Our code and video examples are available at: \href{https://icml-2.github.io/SMD/}{https://icml-2.github.io/SMD/}.

Autonomous driving systems must be tested in dangerous traffic situations before they can be safely used on real roads. However, these situations, such as sudden braking, risky lane changes, or vehicles approaching too quickly, are rare in real-world driving data and unsafe to recreate deliberately. This makes it difficult to know whether an autonomous driving system can handle critical moments. We introduce SMD, a framework that creates realistic driving videos of safety-critical situations from multiple camera views. Instead of only producing simplified simulations or single-camera videos, SMD generates videos that are closer to the type of visual input used by modern autonomous driving systems. It first identifies vehicles that are likely to create risk, then creates realistic vehicle movements that represent dangerous but natural driving situations, and finally turns these movements into multi-view driving videos. These generated videos can be used to test autonomous driving systems under challenging conditions. In our experiments, SMD helps reveal weaknesses in several driving planners by exposing them to difficult scenarios. When the generated videos are added to training, the planners become safer and more robust. Overall, SMD provides a safer and more scalable way to create realistic dangerous driving scenarios for evaluating and improving autonomous driving systems.