On the Anisotropy of Score-Based Generative Models

We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis suggests that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models.

Modern generative systems often work by starting from random noise and gradually refining it into structured data. Even when two such systems are trained in the same way, their internal design can make certain patterns easier to learn than others. This paper studies these built-in preferences. Specifically, we introduce a framework for identifying, before training, the kinds of structure that a generative architecture is naturally more or less likely to learn. By characterizing these architectural biases, our work gives researchers a new way to understand and evaluate generative models, with the potential to guide future designs toward more reliable, predictable behavior.