RePack then Refine: Efficient Diffusion Transformers with Vision Foundation Models

Semantic-rich features from Vision Foundation Models (VFMs) have been leveraged to enhance Latent Diffusion Models (LDMs). However, raw VFM features are typically high-dimensional and redundant, increasing the difficulty of learning and reducing training efficiency for Diffusion Transformers (DiTs). In this paper, we propose Repack then Refine, a three-stage framework that brings the semantic-rich VFM features to DiT while further accelerating learning efficiency. Specifically, the RePack module projects the high-dimensional features onto a compact, low-dimensional manifold. This filters out the redundancy while preserving essential structural information. A standard DiT is then trained for generative modeling on this highly compressed latent space. Finally, to restore the high-frequency details lost due to the compression in RePack, we propose a Latent-Guided Refiner, which is trained lastly for enhancing the image details. On ImageNet-1K, RePack-DiT-XL/1 achieves an FID of 1.82 in only 64 training epochs. With the Refiner module, performance further improves to an FID of 1.65, significantly surpassing latest LDMs in terms of convergence efficiency. Our results demonstrate that packing VFM features, followed by targeted refinement, is a highly effective strategy for balancing generative fidelity with training efficiency. Source code is publicly available at https://github.com/guanfangdong/RePack-then-Refine.

Modern AI image generators often rely on compact internal representations of images to make training faster and more effective. Recently, researchers have started using powerful vision foundation models to provide richer image representations. These representations contain useful high-level information about objects and scenes, but they are often very large and redundant, which makes the image generator harder and slower to train. This paper proposes RePack then Refine, a three-stage framework for more efficient image generation. First, RePack compresses the rich but high-dimensional features from a vision foundation model into a much smaller representation that keeps the most important semantic information. Second, a diffusion transformer is trained to generate images in this compact space, which makes learning faster. Third, a refiner restores fine visual details, such as textures, that may be lost during compression. On ImageNet-1K image generation, our method achieves strong visual quality with far fewer training epochs than many existing approaches. In particular, RePack reaches an FID score of 1.82 after only 64 training epochs, and adding the refiner further improves the score to 1.65. These results show that compressing vision foundation model features before generation, then separately refining image details, is an effective way to balance image quality and training efficiency.