Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

Soft labels from teacher models are a de facto practice for knowledge transfer and large-scale dataset distillation (e.g., SRe$^2$L, LPLD). However, when we limit the number of crops per image to reduce the substantial cost of storing precomputed soft labels, these methods suffer severely from local semantic drift: visually ambiguous crops can cause soft supervision to deviate from the image-level ground-truth semantics, leading to persistent errors and a train–test distribution mismatch. We revisit the overlooked role of hard labels and show that, when properly integrated, they can act as a content-invariant semantic anchor that calibrates such drift. We theoretically analyze the emergence of drift under sparse soft-label supervision and demonstrate that hybridizing hard and soft labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which uses hard labels as intermediate corrective signals while preserving the fine-grained benefits of soft labels. Extensive experiments on dataset distillation and large-scale classification benchmarks show consistent generalization improvements. On ImageNet-1K, our method achieves 42.7% accuracy with only 285M soft-label storage (reduces by 100$\times$), outperforming prior state-of-the-art LPLD by 9.0%. Code is available at https://github.com/Jiacheng8/HALD.

Training modern AI image classifiers often relies on enormous datasets containing millions of pictures, which is slow and expensive. To make this manageable, researchers have developed "dataset distillation", a way to compress huge collections of images into a tiny synthetic substitute that trains models almost as well. The catch is that these compact datasets depend on detailed "soft" guidance labels from a teacher model, and storing those labels can take up more space than the images themselves, sometimes 30 to 40 times more. A natural fix is to keep fewer guidance labels per image, but we found this introduces a hidden problem: when a model only sees small cropped patches of an image, the guidance can drift away from what the picture actually shows, for example, a zoomed-in patch of a cat may look more like a rabbit. We call this local semantic drift. Our solution is to bring back the simple, original class labels (like "cat" or "rabbit") as anchors during a middle stage of training, correcting the drift before fine-tuning resumes. This cuts label storage by up to 100× while substantially improving accuracy on large benchmarks like ImageNet.