Timezone: »

 
Oral
Cones: Concept Neurons in Diffusion Models for Customized Generation
Zhiheng Liu · Ruili Feng · Kai Zhu · Yifei Zhang · Kecheng Zheng · Yu Liu · Deli Zhao · Jingren Zhou · Yang Cao

Tue Jul 25 09:42 PM -- 09:50 PM (PDT) @ Ballroom A
in Oral A4 Diffusion »

Human brains respond to semantic features of presented stimuli with different neurons. This raises the question of whether deep neural networks admit a similar behavior pattern. To investigate this phenomenon, this paper identifies a small cluster of neurons associated with a specific subject in a diffusion model. We call those neurons the concept neurons. They can be identified by statistics of network gradients to a stimulation connected with the given subject. The concept neurons demonstrate magnetic properties in interpreting and manipulating generation results. Shutting them can directly yield the related subject contextualized in different scenes. Concatenating multiple clusters of concept neurons can vividly generate all related concepts in a single image. Our method attains impressive performance for multi-subject customization, even four or more subjects. For large-scale applications, the concept neurons are environmentally friendly as we only need to store a sparse cluster of int index instead of dense float32 parameter values, reducing storage consumption by 90% compared with previous customized generation methods. Extensive qualitative and quantitative studies on diverse scenarios show the superiority of our method in interpreting and manipulating diffusion models.

[ PDF] 

Author Information

Zhiheng Liu (University of Science and Technology of China)
Ruili Feng (USTC)
Kai Zhu (University of Science and Technology of China)
Yifei Zhang (Shanghai Jiao Tong University)
Kecheng Zheng (Ant Research)
Yu Liu (Alibaba Group)
Deli Zhao (Alibaba Group)
Jingren Zhou (Alibaba Group)
Yang Cao (University of Science and Technology of China)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors