Improving Classifier-Free Guidance of Flow Matching via Manifold Projection

Classifier-free guidance (CFG) is a widely used technique for controllable generation in diffusion and flow-based models. Despite its empirical success, CFG relies on a heuristic linear extrapolation that is often sensitive to the guidance scale. In this work, we provide a principled interpretation of CFG through the lens of optimization. We demonstrate that the velocity field in flow matching corresponds to the gradient of a sequence of smoothed distance functions, which guides latent variables toward the scaled target image set. This perspective reveals that the standard CFG formulation is an approximation of this gradient, where the prediction gap, the discrepancy between conditional and unconditional outputs, governs guidance sensitivity. Leveraging this insight, we reformulate the CFG sampling as a homotopy optimization with a manifold constraint. This formulation necessitates a manifold projection step, which we implement via an incremental gradient descent scheme during sampling. To improve computational efficiency and stability, we further enhance this iterative process with Anderson Acceleration without requiring additional model evaluations. Our proposed methods are training-free and consistently refine generation fidelity, prompt alignment, and robustness to the guidance scale. We validate their effectiveness across diverse benchmarks, demonstrating significant improvements on large-scale models such as DiT-XL-2-256, Flux, and Stable Diffusion 3.5. Code is available at https://github.com/LeonSuZhengYi/CFG-MP.

Modern image generators often need guidance from a class label or text prompt to produce the desired content. A common technique for this is classifier-free guidance, which improves controllability by combining two model predictions during sampling. However, its performance depends heavily on a guidance strength parameter: if the value is too small, the image may not follow the prompt well, while if it is too large, the image may contain artifacts or lose diversity. This paper explains classifier-free guidance from an optimization perspective. Our analysis shows that, in flow matching models, guidance can be understood as moving generated samples toward the set of images that match the condition. It also identifies the difference between conditional and unconditional predictions as a key reason why guidance strength is sensitive. Based on this insight, we propose CFG-MP, which adds a projection step during sampling to reduce this prediction difference. We further introduce CFG-MP+, which uses Anderson Acceleration to make this projection more stable and efficient. The method is training-free and can be applied to existing models. Experiments on DiT-XL-2-256, Flux, and Stable Diffusion 3.5 show that it improves image quality, prompt alignment, and robustness to the choice of guidance strength.