When Do Diffusion Models Learn to Generate Multiple Objects?

Text-to-image diffusion models achieve impressive visual fidelity, yet they remain unreliable in multi-object generation. Despite extensive empirical evidence of these failures, the underlying causes remain unclear. We begin by asking how much of this limitation arises from the data itself. To disentangle data effects, we consider two regimes across different dataset sizes: (1) concept generalization, where each individual concept is observed during training under potentially imbalanced data distributions, and (2) compositional generalization, where specific combinations of concepts are systematically held out. To study these regimes, we introduce mosaic (Multi-Object Spatial relations, AttrIbution, Counting), a controlled framework for dataset generation. By training diffusion models on mosaic, we find that scene complexity plays a dominant role rather than concept imbalance, and that counting is uniquely difficult to learn in low-data regimes. Moreover, compositional generalization collapses as more concept combinations are held out during training. These findings highlight fundamental limitations of diffusion models and motivate stronger inductive biases and data design for robust multi-object compositional generation.

Today’s image-generation models can make very realistic pictures, but they still make surprising mistakes when asked to create several objects at once. For example, they may draw the wrong number of objects, give the wrong color to an object, or place objects in the wrong position. We investigate why this happens by studying the training data that these models learn from. We create a controlled benchmark where we can generate images with exact labels for object counts, colors, and spatial relations. This allows us to test two questions separately: whether the model can learn each individual concept when examples are limited or imbalanced, and whether it can combine known concepts in new ways that were not seen during training. We find that multi-object generation is especially difficult when scenes become more complex. Counting objects is particularly unstable when training data is limited, and models struggle more as they must generate more unseen combinations of known concepts. These patterns also appear in more realistic settings. Our results show that reliable image generation requires better mechanisms for handling multiple objects and their relationships.