Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation

Diffusion models have achieved state-of-the-art performance in generating images, audio, and video, but their adaptation to text remains challenging due to its discrete nature. Prior approaches either apply Gaussian diffusion in continuous latent spaces, which inherits semantic structure but struggles with token decoding, or operate in categorical simplex space, which respect discreteness but disregard semantic relation between tokens. In this paper, we propose Smoothing Diffusion on Token Embeddings (Smoothie), a novel diffusion method that combines the strengths of both approaches by progressively smoothing token embeddings based on semantic similarity. This technique enables gradual information removal while maintaining a natural decoding process. Experimental results on several sequence-to-sequence and unconditional generation tasks demonstrate that Smoothie outperforms existing diffusion-based models in generation quality. Furthermore, ablation studies show that our proposed diffusion space yields better performance than both the standard embedding space and the categorical simplex.

Modern AI systems can generate strikingly realistic images, audio, and video using a technique called diffusion, which works by gradually adding random noise to data and then teaching a model to reverse the process. Applying this idea to text has been tricky because words are discrete objects rather than smooth signals like pixels. To solve this problem, the words are usually converted to their vector representations. The choice of the representation type is very important. Existing approaches face a trade-off: some methods preserve word meanings but cannot naturally convert the representations back into actual words, while others handle the word-conversion cleanly but ignore semantic relationships between words during noising. This paper introduces Smoothie, which gets the best of both worlds. Instead of adding random noise directly to words, it gradually "blurs" each word into its semantically similar neighbours — so "joyful" first drifts toward words like "happy" before dissolving into unrelated terms. Across tasks like summarization, paraphrasing, question generation and detoxification, Smoothie outperforms previous diffusion-based text models.