DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery

Dataset distillation aims to synthesize a compact proxy dataset that is unreadable or non-raw from the original dataset for privacy protection and highly efficient learning. However, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning specific patterns that overfit on a prior architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this, we propose a novel dual-stage distillation framework called ${\textbf{DIVER}}$, which leverages the pre-trained diffusion model to dive deeper into $\textbf{DI}$stilled data $\textbf{V}$ia $\textbf{E}$xpressive semantic $\textbf{R}$ecovery, an entire process of semantic inheritance, guidance, and fusion. Semantic inheritance distills high-level semantics of abstract distilled images into the latent space to filter out architecture-specific ``noise" and retain the intrinsic semantics. Furthermore, semantic guidance improves the preservation of the original semantics by directing the reverse procedure. Finally, semantic fusion is designed to provide semantic guidance only during the concrete phase of the reverse process, preventing semantic ambiguity and artifacts while maintaining the guidance information. Extensive experiments validate the effectiveness and efficiency of our method in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256$\times$256) with only 4 GB of GPU memory usage.

Modern AI models often require huge amounts of training data, which can be expensive to store, share, and process. Dataset distillation aims to solve this problem by compressing a large dataset into a much smaller synthetic one that still preserves the important knowledge needed for learning. However, existing methods often create compressed data that works well only for a specific neural network architecture and generalizes poorly to others. In this work, we propose DIVER, a new framework that uses pre-trained diffusion models to better recover and preserve the underlying semantic information hidden in distilled data. DIVER first extracts high-level semantic representations from the compressed images, then uses these semantics to guide image reconstruction while avoiding noisy or misleading details. This process helps the distilled data remain effective across different model architectures. Experiments on standard image benchmarks show that DIVER substantially improves the quality and transferability of distilled datasets while remaining computationally efficient, requiring only modest GPU memory.