Timezone: »

Your Diffusion Model is Secretly a Zero-Shot Classifier
Alexander Li · Mihir Prabhudesai · Shivam Duggal · Ellis Brown · Deepak Pathak
Event URL: https://openreview.net/forum?id=Ck3yXRdQXD »

The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. However, almost all use cases so far have solely focused on sampling. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. We also find that our diffusion-based approach has stronger multimodal relational reasoning abilities than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Even though these models are trained with weak augmentations and no regularization, they approach the performance of SOTA discriminative classifiers. Overall, our results are a step toward using generative over discriminative models for downstream tasks

Author Information

Alexander Li (Carnegie Mellon University)
Mihir Prabhudesai (Carnegie Mellon University)
Shivam Duggal (Uber ATG)
Ellis Brown (Carnegie Mellon University)
Ellis Brown

I am an incoming PhD student at NYU advised by Saining Xie and collaborating with Rob Fergus. I just finished a Master’s in Computer Science at CMU, where I was a graduate student researcher advised by Deepak Pathak and Alyosha Efros. Previously I was a founding engineer at BlackRock AI Labs, where I was fortunate to work with Mykel Kochenderfer, Stephen Boyd, and Trevor Hastie on projects in ML, optimization, and big data applied to finance. Before that, I studied computer science and mathematics at Vanderbilt.

Deepak Pathak (Carnegie Mellon University)

More from the Same Authors