CSD: Content-aware Speculative Decoding for Efficient Image Generation

Speculative decoding (SD) has emerged as a key solution to accelerate the inference of autoregressive models. However, in the field of image generation, it faces the challenge of low acceptance rates, and directly relaxing its criteria leads to degradation in image quality. In this paper, we propose a novel content-aware speculative decoding algorithm, termed CSD, which integrates an entropy-based probability relaxation mechanism with an optimal resampling strategy to enhance the inference efficiency for autoregressive image generation. By leveraging the informational uncertainty inherent in different regions of an image, CSD dynamically adjusts the acceptance probability of candidate tokens, increasing the acceptance rate in low-detail areas to accelerate generation. Moreover, a distribution alignment filter is introduced to ensure the output distribution to be aligned with the target model, which significantly improves the generative quality. Experiments conducted on Lumina-mGPT and Janus-Pro demonstrate that the superiority of the proposed CSD. Our source code is available at https://github.com/aderfebr/CSD.

Current autoregressive image generation models typically adopt a "token-by-token" approach to render images, which is extremely slow. While a promising acceleration technique called speculative decoding (SD) exists, it often struggles in image generation tasks: candidate tokens are either rejected at a high rate, or image quality is sacrificed in pursuit of speed. We identify the root cause as the "one-size-fits-all" nature of existing methods—they treat simple regions and complex textures identically. To solve this, we design a content-aware system. Our method dynamically adjusts how aggressively the model predicts ahead, based on the complexity of each image region. In simple zones, it permits faster, riskier guesses; in detailed zones, it remains cautious and precise. Our experiments show this approach accelerates leading autoregressive image generators by up to 4.3 times. Importantly, this speedup comes without compromising final image quality or requiring extra training. This makes powerful image generation models far more practical for real-time design, on-device applications, and other latency-sensitive scenarios.