VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding

Current Video Large Language Models (Video LLMs) typically encode frames via a vision encoder and employ an autoregressive (AR) LLM for understanding and generation. However, this AR paradigm inevitably faces a dual efficiency bottleneck: strictly unidirectional attention compromises understanding efficiency by hindering global spatiotemporal aggregation, while serial decoding restricts generation efficiency. To address this, we propose VidLaDA, a Video LLM based on Diffusion Language Models (DLMs) that leverages bidirectional attention to unlock comprehensive spatiotemporal modeling and decode tokens in parallel. To further mitigate the computational overhead of diffusion decoding, we introduce MARS-Cache, an acceleration strategy that prunes redundancy by combining asynchronous visual cache refreshing with frame-wise chunk attention. Experiments show VidLaDA rivals state-of-the-art AR baselines (e.g., Qwen2.5-VL and LLaVA-Video) and outperforms DLM baselines, with MARS-Cache delivering over 12x speedup without compromising accuracy. Code and checkpoints are open-sourced at https://github.com/ziHoHe/VidLaDA.

Video large language models help computers understand and describe videos for applications such as visual analysis, education, and digital assistants. Most existing systems encode video frames with a visual model and then use an autoregressive language model, which processes and generates text from left to right. This design has two limitations: causal attention can make it harder to combine information across the whole video, and step-by-step decoding makes generation inherently sequential. To address these issues, we introduce VidLaDA, a video large language model based on diffusion language models. VidLaDA uses bidirectional attention to better capture spatial and temporal relationships throughout the video, and it can generate multiple tokens in parallel. We also propose MARS-Cache, an acceleration strategy that reduces redundant computation during diffusion decoding. Experiments show that VidLaDA is competitive with strong autoregressive video language models and outperforms diffusion-based baselines, while MARS-Cache achieves over 12x speedup without reducing accuracy. This suggests that diffusion language models are a promising direction for efficient and capable video understanding.