Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models

Despite remarkable progress in text-to-image diffusion models, accurately generating the specified number of objects remains a persistent challenge. We identify the initial noise as a primary determinant of spatial layout formation, with early-stage cross-attention serving as the key mechanism that mediates the propagation of noise-induced structures throughout the denoising process. We characterize this phenomenon as \textbf{\textit{Noise-Induced Layout Prior}}. Leveraging this insight, we propose a novel training-free framework for object counting in diffusion models. Our approach consists of two key components: (1) a \emph{Count-Aware Noise Adjustment Strategy}, which explicitly manipulates the initial latent noise to align layout formation with the target object count, and (2) an \emph{Attention-Guided Layout Consistency Strategy}, which performs test-time optimization on early-stage cross-attention to further stabilize layout formation during denoising. Extensive experiments on both single-category and multi-category benchmarks demonstrate that our method consistently outperforms strong diffusion baselines and state-of-the-art object count control methods in terms of counting accuracy and image quality. Code Release: \url{https://github.com/lxlong1201/MitigateNoisePrior}.

Despite the remarkable progress of text-to-image diffusion models, accurately generating a specified number of objects remains a persistent challenge. We observe that the initial noise strongly governs the spatial layout and often dominates object count control throughout the denoising process. We characterize this phenomenon as the Noise-Induced Layout Prior. To address this issue, we propose a novel training-free framework that intervenes in object arrangement during the early stages of denoising. Specifically, we first introduce a Count-Aware Noise Adjustment Strategy to modify the initial latent noise according to the target object count. We further propose an Attention-Guided Layout Consistency Strategy, which performs test-time optimization on early-stage cross-attention maps to enhance layout stability during denoising. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches in object count control. More importantly, our findings reveal the critical role of early-stage noise initialization and attention dynamics, offering new insights into controlling complex compositional properties in generative models.