Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

Dataset distillation (DD) compresses a large training set into a small synthetic set for efficient training, but most DD methods optimize only clean accuracy and leave robustness uncontrolled. Recent robust DD methods improve robustness, yet they often suffer from a poor accuracy–robustness trade-off because they (i) treat all adversarially perturbed examples uniformly, despite robust risk being dominated by near-zero robust margins, and (ii) do not explicitly increase inter-class separation in the decision boundary where attacks concentrate. We present Contrastive Curriculum for Robust Dataset Distillation (C$^2$R), a framework that couples an attack-aware curriculum with a contrastive robustness objective. From a robust-margin perspective, we derive a \emph{perturbation score} that approximates each sample’s robust hinge, enabling a curriculum that prioritizes the smallest-margin adversaries that most directly drive robust error. In parallel, a class-balanced contrastive robustness loss enforces adversarial invariance while explicitly widening boundary separation across classes. Experiments on CIFAR-10/100, Tiny-ImageNet, and multiple ImageNet-1K subsets under six attacks show that C$^2$R achieves the best robust accuracy, outperforming prior robust DD by 2.8% on average.

Training modern AI models often requires very large datasets, which are costly to store and reuse. Dataset distillation reduces this cost by replacing a large dataset with a much smaller artificial one, but models trained on such data can still be fragile when images are slightly changed. This paper makes these small artificial datasets more reliable. Instead of treating all examples equally, our method focuses on the ones most likely to cause mistakes. It also helps the model better separate different categories, so small changes are less likely to confuse it. Experiments on several image datasets show that our method improves reliability while keeping normal accuracy competitive.