Prism-MoE: Efficient Dense-to-MoE Conversion for Visual Autoregressive Generation

Scaling up visual autoregressive models improves generation quality but incurs substantial inference costs. Mixture-of-Experts (MoE) architectures mitigate this issue through sparse activation and have proven effective in large language models. However, training MoE models from scratch remains prohibitively expensive, and dense-to-MoE conversion for visual autoregressive models is still underexplored. To enable low-cost and high-quality dense-to-MoE conversion, we propose Prism-MoE, an efficient framework for transforming pretrained dense visual autoregressive models into sparse MoE models. Prism-MoE consists of two key components. First, we introduce trajectory-consistent Initialization, which formulates expert initialization as a principled decomposition problem and preserves the generation trajectory of pretrained models. Second, we propose a confidence-adaptive sparse fine-tuning framework that aligns expert specialization with the information density of visual tokens via confidence-aware routing supervision. Experiments show that Prism-MoE achieves dense-to-MoE conversion with less than 10\% of the standard training budget, while maintaining generation quality comparable to dense baselines with only 37.5\% active parameters.

Modern image generation models can produce high-quality images, but they are increasingly large and expensive to run. One way to reduce this cost is to use a “mixture of experts” design, where only part of the model is activated for each input. However, training such models from scratch is very costly. This paper studies a more practical problem: how to convert an already trained image generation model into a more efficient mixture-of-experts model. This conversion is especially challenging for autoregressive image generation, where an image is produced step by step. If the converted model makes small mistakes in early steps, these errors can accumulate and harm the final image. We propose Prism-MoE, a two-stage method that first initializes the efficient model to closely follow the behavior of the original model, and then lightly fine-tunes it so that different parts of the model specialize in different visual patterns. Experiments on both class-conditioned and text-conditioned image generation show that Prism-MoE can substantially reduce the amount of model computation used during generation while maintaining image quality close to the original dense model. This makes high-quality image generation more efficient without requiring the high cost of training a new mixture-of-experts model from scratch.