Factored Classifier-Free Guidance

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose Factored Classifier-Free Guidance (FCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. FCFG complements recent advances in classifier-free guidance and can be seamlessly extended to advanced guidance schemes such as CFG++ and APG. Our experiments demonstrate that FCFG significantly improves the axiomatic soundness of inferred counterfactuals across both natural and medical image datasets, mitigating spurious amplification effects, and enhancing counterfactual reversibility.

Diffusion models can generate realistic counterfactual images: images that show how an input might look if a specific attribute were changed. For example, a counterfactual medical image might ask how a scan would look if a particular finding were present or absent. These images are useful for causal reasoning because they let us simulate how an observed image might change under a hypothetical intervention, while preserving factors that are not causally affected by that intervention. A common technique called classifier-free guidance strengthens the conditioning signal during image generation. However, standard classifier-free guidance uses one global guidance weight for all conditioning attributes. In counterfactual generation, this can be problematic: increasing the guidance strength for the intended intervention can also amplify unrelated attributes that should remain invariant. We propose Factored Classifier-Free Guidance, which separates conditioning attributes into semantic or causal groups and assigns different guidance weights to each group. This allows stronger guidance for intervened attributes while reducing unintended guidance on invariant attributes. Our method works at inference time and does not require retraining the diffusion model. We evaluate it on face, mammography, and chest X-ray datasets, showing that it reduces unintended attribute changes, improves reversibility, and better preserves identity.