Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner

Diffusion language models, especially masked discrete diffusion models, have achieved great success recently. While there are some theoretical and primary empirical results showing the advantages of latent reasoning with looped transformers or continuous CoT, continuous diffusion models typically underperform their discrete counterparts. In this paper, we argue that diffusion language models do not necessarily need to be in the discrete space. In particular, we prove that continuous diffusion models have stronger expressivity than discrete diffusions and looped transformers. We attribute the contradiction between the theoretical expressiveness and empirical performance to their practical trainability: while continuous diffusion provides intermediate supervision that looped transformers lack, they are harder to generate and decode tokens in the continuous representation space compared with discrete states. We therefore propose Coevolutionary Continuous Discrete Diffusion (CCDD), which defines a joint multimodal diffusion process on the union of a continuous representation space and a discrete token space, leveraging a single model to simultaneously denoise in the joint space. By combining two modalities, CCDD is expressive with rich semantics in the latent space, as well as good trainability and sample quality with the help of explicit discrete tokens. We also propose effective architectures and advanced training/sampling techniques for CCDD, which reveals strong empirical performance in extensive language modeling experiments on real-world tasks.

Imagine an AI trying to solve a complex problem or write a story. Normally, AI models build sentences by picking words one by one from a rigid dictionary (a "discrete" approach). Recently, some researchers have tried a more fluid "continuous" approach, where the AI processes abstract concepts instead of exact words. This fluid method is fantastic for deep, complex "thinking"—allowing the AI to perform "latent reasoning" (pondering ideas behind the scenes before speaking). However, it often struggles to accurately translate those abstract thoughts back into readable, perfectly accurate text. This paper introduces a new method called Coevolutionary Continuous Discrete Diffusion (CCDD). Instead of forcing the AI to choose between being a deep thinker (using fluid concepts) or a clear speaker (using exact words), CCDD combines both simultaneously. The AI works in a joint environment, refining its deep, hidden reasoning and its actual word choices at the very same time using a single model. As a result, the AI gets the best of both worlds: it can perform complex, behind-the-scenes reasoning to tackle difficult questions, while still reliably producing excellent, clear human language.