ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving

Safety-critical scenarios are central to evaluating autonomous driving systems, yet their rarity in naturalistic logs makes simulation-based stress testing indispensable. Most scenario generation methods treat surrounding agents as adversaries, but they either (i) induce failures without explicitly modeling vehicle-road physical limits, yielding visually extreme yet physically unsolvable crashes, or (ii) enforce physical feasibility or policy feasibility in isolation, which can over-focus on aggressive maneuvers or remain tied to a controller-dependent capability boundary. We propose ScenePilot, a feasibility-guided, boundary-driven framework that targets the boundary band: scenarios that are physically solvable in principle yet still cause the deployed autonomy stack to fail. We formulate generation as constrained multi-objective reinforcement learning, combining an RSS-derived physical-feasibility score $\sigma$ with an online-learned AV-risk predictor $\Phi$, and introduce step-level feasibility-aware shielding to keep exploration near the feasibility boundary while avoiding infeasible artifacts. Experiments on SafeBench with multiple planners show that ScenePilot yields substantially higher collision rates (+6.2 percentage points) while preserving physical validity, and that adversarial fine-tuning on these boundary-band scenarios consistently reduces downstream crash rates. The code is available at https://github.com/QiyuRuan/ScenePilot.

Autonomous driving systems need to be tested in difficult and potentially dangerous situations before they can be safely deployed. However, such situations are rare in real-world driving data, and testing them directly on public roads would be unsafe. Simulation offers a safer way to create and study these challenging cases. In this work, we ask how to generate driving scenarios that are both realistic and useful for testing. Some existing methods create crashes that are too extreme, where even an ideal driver could not avoid the accident. Other methods focus too much on the weakness of one particular driving system. We propose ScenePilot, a method that searches for difficult but still physically avoidable driving situations. These cases are especially valuable because they reveal where an autonomous driving system fails even though a safe solution should still exist. We test ScenePilot in simulation with several driving systems. Our results show that it creates more meaningful safety-critical scenarios and that using these scenarios for further training can help reduce future crashes.