Multimodal Latent Language Modeling with Next-Token Diffusion

Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video). In this work, we propose Latent Language Modeling (LatentLM), which seamlessly integrates continuous and discrete data using causal Transformers. Specifically, we employ a variational autoencoder (VAE) to represent continuous data as latent vectors and introduce next-token diffusion for autoregressive generation of these vectors. Additionally, we develop $\sigma$-VAE to address the challenges of variance collapse, which is crucial for autoregressive modeling. Extensive experiments demonstrate the effectiveness of LatentLM across various modalities. In image generation, LatentLM sis competitive with or outperforms DiT-style baselines under matched unified settings. When integrated into multimodal large language models, LatentLM provides a general-purpose interface that unifies multimodal generation and understanding. Experimental results show that LatentLM achieves favorable performance compared to Transfusion and vector quantized models in the setting of scaling up training tokens. In text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2 model in speaker similarity and robustness, while requiring 10 fewer decoding steps. The results establish LatentLM as a highly effective and scalable approach to advance large multimodal models.

Modern AI systems need to handle many types of data, such as text, images, and speech. Text is naturally represented as discrete tokens, while images and speech are continuous signals. Existing methods often either use separate models for different data types or convert continuous data into discrete codes, which can lose information. This paper proposes LatentLM, a unified model that represents images and speech as continuous latent vectors and generates them one by one, similar to how language models generate text tokens. To make this stable, we introduce σ-VAE, which makes the latent space more robust to the small errors that accumulate during autoregressive generation. We evaluate LatentLM on image generation, multimodal understanding and generation, and text-to-speech synthesis. The results show that continuous-token autoregressive modeling can be competitive with strong existing methods while keeping the simplicity and efficiency of causal language models.