DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Driving

End-to-end autonomous driving systems are increasingly integrating Vision-Language Model (VLM) architectures, incorporating text reasoning or visual reasoning to enhance the robustness and accuracy of driving decisions. However, the reasoning mechanisms employed in most methods are direct adaptations from general domains, lacking in-depth exploration tailored to autonomous driving scenarios, particularly within visual reasoning modules. In this paper, we propose a driving world model that performs parallel prediction of latent semantic features for consecutive future frames in the bird’s-eye-view (BEV) space, thereby enabling long-horizon modeling of future world states. We also introduce an efficient and adaptive text reasoning mechanism that utilizes additional social knowledge and reasoning capabilities to further improve driving performance in challenging long-tail scenarios. We present a novel, efficient, and effective approach that achieves state-of-the-art (SOTA) results on the closed-loop Bench2drive benchmark. Codes are available at: https://github.com/hotdogcheesewhite/DeepSight.

Autonomous driving systems need to “look ahead” and understand complex road situations to drive safely, just like human drivers. Current self-driving models often only predict a short moment into the future and struggle with unusual or tricky traffic scenarios, such as construction zones or emergency vehicles. We created DeepSight, a self-driving system that can foresee road conditions for a longer time and make smarter decisions. It uses a simple, efficient way to imagine future road scenes and only activates extra logical thinking when facing difficult situations. Results show DeepSight drives more safely, completes routes more reliably, and reacts wisely to challenging traffic cases without slowing down noticeably.