Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

While recent multimodal large language models (MLLMs) have made impressive strides, they mostly employ a conventional autoregressive architecture as their backbone, leaving significant room for exploring effective and efficient alternatives in architectural design. Meanwhile, recent studies have successfully applied discrete diffusion models to natural language processing, revealing their considerable potential as a promising new approach in this domain. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Our codes are released at GitHub.

Most existing AI systems built to handle multimodal data, like text, images, and speech, employ an autoregressive architecture. This design typically generates tokens strictly from beginning to end, leaving ample room for exploring alternative architectural designs. Recently, discrete diffusion models, an emerging class of generative models, have been successfully applied to natural language modeling. Inspired by these advances, we sought to extend discrete diffusion models to build unified multimodal AI systems. To achieve this, we introduce Omni-Diffusion, a multimodal language model built on a mask-based diffusion model. In contrast to autoregressive models, when generating a multimodal data sample, our model starts with a blank sequence and iteratively fills in the sequence until a complete output is formed. By treating text, images, and speech simply as different types of data, our method can process and generate them seamlessly within a single framework. We tested our model on various tasks and found that this approach exhibits strong multimodal capabilities, matching or even outperforming existing autoregressive models. Furthermore, by predicting multiple parts of the data at once, our model can generate results faster. Overall, this work highlights the significant promise of diffusion models for building the next generation of multimodal AI systems.