Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation

We present Causal-Adapter, a modular framework that adapts frozen text-to-image diffusion backbones for counterfactual image generation. Our method enables causal interventions on target attributes while preserving all other aspects of the image, including the core identity. In contrast to prior approaches that rely on prompt engineering without explicit causal structure, Causal-Adapter leverages structural causal modeling augmented with two attribute regularization strategies: prompt-aligned injection, which aligns causal attributes with textual embeddings for precise semantic control, and a conditioned token contrastive loss to disentangle attribute factors and reduce spurious correlations. Causal-Adapter achieves state-of-the-art results on synthetic and real-world datasets, outperforming other baselines in effectiveness, composition, realism, and minimality. These results demonstrate that the approach enables efficient, robust, and generalizable counterfactual image editing with faithful attribute modification and strong preservation of core identity.

Counterfactual image generation asks a simple but important question: what would an image look like if one meaningful factor were changed? For example, what would a brain scan look like if the ventricle were larger, or what would a face look like if the person were older? Current image editing models can often make visually realistic changes, but they may also alter unrelated details or follow superficial correlations rather than true cause-and-effect relationships. We introduce Causal-Adapter, a lightweight module that helps a frozen text-to-image model perform more faithful counterfactual editing. Instead of retraining the whole image generation model, our method injects causal information about attributes, such as age, gender, beard, or brain volume, into the model through a small adapter. This allows the model to change the intended attribute while preserving the identity and other important image details. Experiments on synthetic scenes, human faces, and medical brain images show that Causal-Adapter produces more accurate, realistic, and causally consistent edits than existing methods. This can make image generation more useful for scientific analysis, medical imaging research, and trustworthy visual reasoning.