MMD Guidance: Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance

Pre-trained diffusion models have emerged as powerful generative priors for both unconditional and conditional sample generation, yet their outputs often deviate from the characteristics of user-specific target data. Such mismatches are especially problematic in domain adaptation tasks, where only a few reference examples are available and retraining the diffusion model is infeasible. Existing inference-time guidance methods can adjust sampling trajectories, but they typically optimize surrogate objectives such as classifier likelihoods rather than directly aligning with the target distribution. We propose MMD Guidance, a training-free mechanism that augments the reverse diffusion process with gradients of the Maximum Mean Discrepancy (MMD) between generated samples and a reference dataset. MMD provides reliable distributional estimates from limited data, exhibits low variance in practice, and is efficiently differentiable, which makes it particularly well-suited for the guidance task. Our framework naturally extends to prompt-aware adaptation in conditional generation models via product kernels. Also, it can be applied with computational efficiency in latent diffusion models (LDMs), since guidance is applied in the latent space of the LDM. Experiments on synthetic and real-world benchmarks demonstrate that MMD Guidance can achieve distributional alignment while preserving sample fidelity. The project code is available at https://github.com/matinamehdizadeh/MMD-Guidance.

Modern diffusion models can generate high-quality images, but their outputs often do not match the specific style or domain a user has in mind. Fine-tuning such models for every new target distribution is costly and often impractical, especially when only a small reference set is available. This paper introduces MMD Guidance, a training-free method that adapts diffusion models during sampling. The method uses Maximum Mean Discrepancy, a kernel-based measure of distributional distance, to steer generated samples toward a user-provided reference set without changing the model’s parameters. It also extends naturally to text-conditioned generation, where guidance can account for both prompt semantics and visual style, and it is implemented efficiently in the latent space of modern diffusion models. Experiments show that MMD Guidance improves alignment to target distributions while preserving image quality and diversity. Overall, the method offers a practical way to personalize diffusion models using only a limited number of examples, without retraining or fine-tuning.