Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression

Dataset pruning (DP) and dataset distillation (DD) fundamentally differ in their outputs: DP selects original image subsets, while DD generates synthetic images. Recently, DD's increasing reliance on original images suggests a convergence of the two directions. To investigate this convergence trend, we propose a unified dataset compression (DC) benchmark. This benchmark reveals an interesting trade-off for soft-label-DD: while soft labels provide valuable information, they can make the distillation process less essential, as distilled images may not always outperform random subsets. In addition, the benchmark reveals that in current stages, dataset pruning outperforms dataset distillation at small dataset sizes. Given these observations, we explore hard-label-DC as a complementary approach that emphasizes image quality while offering substantial storage efficiency. Our PCA (Prune, Combine, and Augment) is the first framework that does not rely on soft labels but instead focuses on image quality. (1) "P'' means selecting easy samples based on dataset pruning metrics, (2) "C'' indicates combining these samples effectively, and (3) "A'' is to apply constrained image augmentation during training.

Training modern AI models requires huge image datasets, which are expensive to store and slow to train on. Researchers have explored two ways to shrink these datasets: dataset pruning, which keeps a small set of the most useful real images, and dataset distillation, which creates a few synthetic images designed to capture the essence of the full dataset. Recently, distillation methods have leaned heavily on a trick called "soft labels", rich answer keys attached to each image that quietly carry most of the useful information. We built a common benchmark to compare pruning and distillation fairly, and found something surprising: once soft labels are doing the heavy lifting, the synthetic images themselves often perform no better than randomly chosen real ones. We also found that at very small dataset sizes, simple pruning beats distillation. Motivated by this, we propose PCA (Prune, Combine, Augment), a method that skips soft labels entirely and instead focuses on choosing and using real images well: pick the clearest examples, combine them smartly, and augment them carefully during training. The result is a simpler, more storage-efficient approach to shrinking datasets without sacrificing model quality.