Visual Counterfactual Explanations with Compositional Generative Models

Generating realistic counterfactual explanations for vision models requires changing the right visual factors while preserving the rest of the image. Existing approaches typically edit pixels or holistic latent codes, often producing entangled changes that obscure what drove the prediction shift. We propose VOCCE, an object-centric counterfactual generation method that operates on slot-based representations and a slot-conditioned diffusion backbone. Given an input image and a target label, VOCCE steers reverse diffusion using classifier guidance on the denoising prediction and gradient-based updates to the object slots, enabling targeted object- or part-level interventions. To prevent off-manifold slot drift and preserve realism, we introduce a Gaussian Mixture Model prior over slot states as a regularizer. Experiments on ClevrTex and CelebA-HQ, plus cross-dataset transfer from FFHQ to CelebA-HQ show that VOCCE improves locality, closeness, and visual fidelity compared to state-of-the-art counterfactual baselines, and a user study indicates preferences for more meaningful and subtle edits.