Mosaic: Unlocking Over 30$\times$ Context Length for Diffusion LLMs Inference via Global Memory Planning and Dynamic Peak Taming

Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive models, leveraging simultaneous denoising to enable global planning and iterative refinement. These properties make dLLMs attractive for long-context generation. However, deploying dLLMs faces a prohibitive memory barrier, as existing inference systems are inefficient for the diffusion paradigm. We observe that current inference systems are misaligned with dLLMs. Unlike autoregressive models, whose memory footprint is dominated by the KV-Cache, dLLMs are bottlenecked by transient activations rematerialized per step. Moreover, generic memory reuse mechanisms lack the global visibility to handle dynamic memory peaks of dLLMs, which alternate between logits and feed-forward networks. To address these challenges, we present Mosaic, a memory-efficient inference system that shifts dLLM execution from local, static management to a global, dynamic paradigm. Mosaic integrates (i) a mask-only logits kernel eliminating redundant activation materialization, (ii) a lazy chunking optimizer using online heuristics to tame dynamic memory peaks, and (iii) a global memory manager leveraging virtual addressing to mitigate memory fragmentation. Evaluations show that Mosaic reduces the memory peak-to-average ratio by 2.71$\times$ on average and increases the maximum inference sequence length on identical hardware by 15.30--32.34$\times$. Crucially, Mosaic is training-free and preserves exact model outputs, while reducing end-to-end latency by 2.5\%--55.4\%. Our code is publicly available at https://github.com/flashserve/Mosaic.

Diffusion-based Large Language Models (dLLMs) are a powerful new type of AI that can generate text by repeatedly refining the entire document, making them excellent for handling very long documents. However, they require massive amounts of computer memory to run, which makes them expensive and difficult to deploy. We found that current systems manage memory inefficiently because they were designed for older, word-by-word AI models. To solve this, we created Mosaic, a new software system specifically designed to handle the unique, unpredictable memory demands of dLLMs. Mosaic intelligently plans and recycles computer memory in real-time. As a result, it allows computers to generate texts that are over 30 times longer using the exact same hardware, speeds up the generation process, and maintains the exact same output quality.