Ultra-Resolution Adaptation with Ease

Text-to-image diffusion models have achieved remarkable progress in recent years. However, training models for high-resolution image generation remains challenging, particularly when training data and computational resources are limited. In this paper, we explore this practical problem from two key perspectives: data and parameter efficiency, and propose a set of key guidelines for ultra-resolution adaptation termed URAE. For data efficiency, we theoretically and empirically demonstrate that synthetic data generated by some teacher models can significantly promote training convergence. For parameter efficiency, we find that tuning minor components of the weight matrices outperforms widely-used low-rank adapters when synthetic data are unavailable, offering substantial performance gains while maintaining efficiency. Additionally, for models leveraging guidance distillation, such as FLUX, we show that disabling classifier-free guidance, i.e., setting the guidance scale to 1 during adaptation, is crucial for satisfactory performance. Extensive experiments validate that URAE achieves comparable 2K-generation performance to state-of-the-art closed-source models like FLUX1.1 [Pro] Ultra with only 3K samples and 2K iterations, while setting new benchmarks for 4K-resolution generation. Codes are available here.

Text-to-image diffusion models have improved rapidly, but training them to generate high-resolution images remains challenging, especially when data and computational resources are limited. Our work addresses this problem by focusing on data efficiency and parameter-efficient fine-tuning, leading to several practical insights. Based on our findings, we provide guidelines for training high-performing models under resource constraints. Especially, we show that synthetic data generated by teacher models can significantly accelerate training process and enhance stability. When synthetic data is not available, we find that tuning minor parts of model leads to better performance than widely-used low-rank adaptation methods, while still maintaining efficiency. Finally, for advanced models that incorporate additional guidance from a more powerful teacher model, such as FLUX, we demonstrate that turning off classifier-free guidance is crucial for achieving satisfactory results. In summary, our proposed guidelines lowers the barrier for developing state-of-the-art text-to-image models, making advanced image generation accessible across a wider range of practical settings.