Timezone: »

 
Poster
Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation
Jiaming Song · Qinsheng Zhang · Hongxu Yin · Morteza Mardani · Ming-Yu Liu · Jan Kautz · Yongxin Chen · Arash Vahdat

Tue Jul 25 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #211
in Poster Session 2 »

We consider guiding denoising diffusion models with general differentiable loss functions in a plug-and-play fashion, enabling controllable generation without additional training. This paradigm, termed Loss-Guided Diffusion (LGD), can easily be integrated into all diffusion models and leverage various efficient samplers. Despite the benefits, the resulting guidance term is, unfortunately, an intractable integral and needs to be approximated. Existing methods compute the guidance term based on a point estimate. However, we show that such approaches have significant errors over the scale of the approximations. To address this issue, we propose a Monte Carlo method that uses multiple samples from a suitable distribution to reduce bias. Our method is effective in various synthetic and real-world settings, including image super-resolution, text or label-conditional image generation, and controllable motion synthesis. Notably, we show how our method can be applied to control a pretrained motion diffusion model to follow certain paths and avoid obstacles that are proven challenging to prior methods.

[ PDF] 

Author Information

Jiaming Song (NVIDIA)
Qinsheng Zhang (Georgia Institution of Technology)
Hongxu Yin (NVIDIA)
Morteza Mardani (Stanford University)
Ming-Yu Liu (NVIDIA)
Jan Kautz (NVIDIA)
Yongxin Chen (Georgia Institute of Technology)
Arash Vahdat (NVIDIA Research)

More from the Same Authors