Timezone: »
Recent studies have demonstrated that gradient matching-based dataset synthesis, or dataset condensation (DC), methods can achieve state-of-theart performance when applied to data-efficient learning tasks. However, in this study, we prove that the existing DC methods can perform worse than the random selection method when taskirrelevant information forms a significant part of the training dataset. We attribute this to the lack of participation of the contrastive signals between the classes resulting from the class-wise gradient matching strategy. To address this problem, we propose Dataset Condensation with Contrastive signals (DCC) by modifying the loss function to enable the DC methods to effectively capture the differences between classes. In addition, we analyze the new loss function in terms of training dynamics by tracking the kernel velocity. Furthermore, we introduce a bi-level warm-up strategy to stabilize the optimization. Our experimental results indicate that while the existing methods are ineffective for fine-grained image classification tasks, the proposed method can successfully generate informative synthetic datasets for the same tasks. Moreover, we demonstrate that the proposed method outperforms the baselines even on benchmark datasets such as SVHN, CIFAR-10, and CIFAR-100. Finally, we demonstrate the high applicability of the proposed method by applying it to continual learning tasks.
Author Information
Saehyung Lee (Seoul National University)
SANGHYUK CHUN (Naver corp.)
Sangwon Jung (Sungkwunkwan university)
Sangdoo Yun ( Clova AI Research, NAVER Corp.)
Sungroh Yoon (Seoul National University)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: Dataset Condensation with Contrastive Signals »
Thu. Jul 21st 03:35 -- 03:40 PM Room Ballroom 1 & 2
More from the Same Authors
-
2023 : De-stereotyping Text-to-image Models through Prompt Tuning »
Eunji Kim · Siwon Kim · Chaehun Shin · Sungroh Yoon -
2023 Poster: Improving Visual Prompt Tuning for Self-supervised Vision Transformers »
Seungryong Yoo · Eunji Kim · Dahuin Jung · JUNGBEOM LEE · Sungroh Yoon -
2023 Poster: On the Impact of Knowledge Distillation for Model Interpretability »
Hyeongrok Han · Siwon Kim · Hyun-Soo Choi · Sungroh Yoon -
2023 Poster: Probabilistic Concept Bottleneck Models »
Eunji Kim · Dahuin Jung · Sangha Park · Siwon Kim · Sungroh Yoon -
2022 Poster: AutoSNN: Towards Energy-Efficient Spiking Neural Networks »
Byunggook Na · Jisoo Mok · Seongsik Park · Dongjin Lee · Hyeokjun Choe · Sungroh Yoon -
2022 Poster: Dataset Condensation via Efficient Synthetic-Data Parameterization »
Jang-Hyun Kim · Jinuk Kim · Seong Joon Oh · Sangdoo Yun · Hwanjun Song · Joonhyun Jeong · Jung-Woo Ha · Hyun Oh Song -
2022 Spotlight: Dataset Condensation via Efficient Synthetic-Data Parameterization »
Jang-Hyun Kim · Jinuk Kim · Seong Joon Oh · Sangdoo Yun · Hwanjun Song · Joonhyun Jeong · Jung-Woo Ha · Hyun Oh Song -
2022 Spotlight: AutoSNN: Towards Energy-Efficient Spiking Neural Networks »
Byunggook Na · Jisoo Mok · Seongsik Park · Dongjin Lee · Hyeokjun Choe · Sungroh Yoon -
2022 Poster: Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance »
Heeseung Kim · Sungwon Kim · Sungroh Yoon -
2022 Spotlight: Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance »
Heeseung Kim · Sungwon Kim · Sungroh Yoon -
2022 Poster: Confidence Score for Source-Free Unsupervised Domain Adaptation »
Jonghyun Lee · Dahuin Jung · Junho Yim · Sungroh Yoon -
2022 Spotlight: Confidence Score for Source-Free Unsupervised Domain Adaptation »
Jonghyun Lee · Dahuin Jung · Junho Yim · Sungroh Yoon -
2020 Poster: Learning De-biased Representations with Biased Representations »
Hyojin Bahng · SANGHYUK CHUN · Sangdoo Yun · Jaegul Choo · Seong Joon Oh -
2019 : Spotlight »
Tyler Scott · Kiran Thekumparampil · Jonathan Aigrain · Rene Bidart · Priyadarshini Panda · Dian Ang Yap · Yaniv Yacoby · Raphael Gontijo Lopes · Alberto Marchisio · Erik Englesson · Wanqian Yang · Moritz Graule · Yi Sun · Daniel Kang · Mike Dusenberry · Min Du · Hartmut Maennel · Kunal Menda · Vineet Edupuganti · Luke Metz · David Stutz · Vignesh Srinivasan · Timo Sämann · Vineeth N Balasubramanian · Sina Mohseni · Rob Cornish · Judith Butepage · Zhangyang Wang · Bai Li · Bo Han · Honglin Li · Maksym Andriushchenko · Lukas Ruff · Meet P. Vadera · Yaniv Ovadia · Sunil Thulasidasan · Disi Ji · Gang Niu · Saeed Mahloujifar · Aviral Kumar · SANGHYUK CHUN · Dong Yin · Joyce Xu Xu · Hugo Gomes · Raanan Rohekar -
2019 Poster: FloWaveNet : A Generative Flow for Raw Audio »
Sungwon Kim · Sang-gil Lee · Jongyoon Song · Jaehyeon Kim · Sungroh Yoon -
2019 Oral: FloWaveNet : A Generative Flow for Raw Audio »
Sungwon Kim · Sang-gil Lee · Jongyoon Song · Jaehyeon Kim · Sungroh Yoon -
2019 Poster: HexaGAN: Generative Adversarial Nets for Real World Classification »
Uiwon Hwang · Dahuin Jung · Sungroh Yoon -
2019 Oral: HexaGAN: Generative Adversarial Nets for Real World Classification »
Uiwon Hwang · Dahuin Jung · Sungroh Yoon