AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture -of-Transformers for End-to-End Autonomous Driving

Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with layer-wise joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient asynchronous inference over various tasks at different frequencies. Additionally, we explore a VLA-oriented action refiner that further enhances driving performance via diffusion-based fine-tuning. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves state-of-the-art (SOTA) performance compared to existing methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when and to what extent AD-tailored fine-tuning is necessary.

Autonomous driving requires two abilities that are difficult to combine: understanding complex traffic scenes and making timely driving decisions. Large VLMs are strong at understanding the world, but they are often too slow for real-time driving if the entire model is executed at every decision step. This creates a fundamental tension between high-level reasoning and fast control. AutoMoT addresses this tension by rethinking how intelligence should be organized inside a driving model. Rather than relying on a hierarchical pipeline that separates perception, reasoning, decision-making, and planning into loosely connected stages, AutoMoT builds a unified VLA model where understanding and action are integrated within one architecture. Its understanding expert reasons about the driving scene, while its action expert rapidly produces decisions and trajectories. Crucially, the two experts share knowledge through latent representations, allowing driving actions to be guided by the model’s internal understanding of the scene. More importantly, AutoMoT suggests that future driving AI may not need to fully tailor an entire foundation model to the driving domain, nor rely on loosely connected hierarchical pipelines. A more scalable path may be to build unified systems where foundation models focus on understanding the world, specialized action modules learn how to drive, and both are efficiently connected for real-time deployment.