Local Mechanisms of Compositional Generalization in Conditional Diffusion

Conditional diffusion models appear capable of compositional generalization, i.e., generating convincing samples for out-of-distribution combinations of conditioners, but the mechanisms underlying this ability remain unclear. To make this concrete, we study length generalization, the ability to generate images with more objects than seen during training. In a controlled CLEVR setting (Johnson et al., 2017), we find that length generalization is achievable in some cases but not others, suggesting that models only sometimes learn the underlying compositional structure. We then investigate locality as a structural mechanism for compositional generalization. Prior works proposed score locality as a mechanism for creativity in unconditional diffusion models (Kamb & Ganguli, 2024; Niedoba et al., 2024), but did not address flexible conditioning or compositional generalization. In this paper, we prove an exact equivalence between a specific compositional structure (conditional projective composition) (Bradley et al., 2025) and scores with sparse dependencies on both pixels and conditioners (local conditional scores). This theory also extends to compositions of concepts (such as style+content) in feature-space. We validate our theory empirically: CLEVR models that succeed at length generalization exhibit local conditional scores, while those that fail do not. Furthermore, we show that a causal intervention explicitly enforcing local conditional scores enables length generalization in a previously failing model. Finally, we investigate feature-space compositionality in color-conditioned CLEVR, and find preliminary evidence of compositional structure and corresponding local mechanisms in SDXL.

Why do diffusion models sometimes succeed at generating novel combinations of concepts, such as "a cat eating sushi with chopsticks in the style of Van Gogh", and sometimes fail? This paper studies a controlled version of the problem: can a diffusion model trained on images with 1–3 objects generate images with 6 or 9 objects? The answer depends on whether the model learns "local conditional scores" — meaning each image region attends only to its nearby, relevant conditioning signals. We show that this locality property is equivalent to a compositional factorization of the distribution, then confirm experimentally that models exhibiting it length-generalize while those lacking it do not. A direct causal intervention — enforcing locality through architecture — restores generalization in a previously failing model. We also find evidence of analogous structures in SDXL's learned feature space, connecting the theory to real-world text-to-image models.