Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing

Modern Latent Diffusion Models (LDMs) typically operate in low-level Variational Autoencoder (VAE) latent spaces that are primarily optimized for pixel-level reconstruction. To unify vision generation and understanding, a burgeoning trend is to adopt high-dimensional features from representation encoders as generative latents. However, we empirically identify two fundamental obstacles in this paradigm: (1) the discriminative feature space lacks compact regularization, making diffusion models prone to off-manifold latents that lead to inaccurate object structures; and (2) the encoder’s inherently weak pixel-level reconstruction hinders the generator from learning accurate fine-grained geometry and texture. In this paper, we propose a systematic framework to adapt understanding-oriented encoder features for generative tasks. We introduce a semantic–pixel reconstruction objective to regularize the latent space, enabling the compression of both semantic information and fine-grained details into a highly compact representation (96 channels with $16\times$ spatial downsampling). This design allows the latent space to remain semantically rich while achieving state-of-the-art image reconstruction, and keeps it compact enough for accurate generation. Leveraging this representation, we design a unified text-to-image (T2I) and image editing model. Across diverse generation spaces, our approach achieves state-of-the-art reconstruction, faster convergence, and substantial gains in both T2I and editing tasks, demonstrating that representation encoders can be effectively adapted into robust generative components.

AI systems that create or edit images need a compact internal description of each image. Existing systems often use descriptions that preserve visual details but do not capture enough high-level meaning, while image understanding systems capture meaning but often lose fine details such as texture, text, and facial structure. This mismatch makes it difficult to build one image representation that works well for both understanding and generation. We study why this happens and propose PS-VAE, a method that adapts strong image understanding models so they can also support high-quality image generation and editing. The key idea is to preserve both semantic information, such as objects and layout, and fine visual details, such as geometry and texture, in a compact representation. As a result, our method improves image reconstruction, text-to-image generation, and instruction-based image editing. This work is a step toward image AI systems that can both understand visual content and generate or edit it reliably.