Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation

Text-to-image (T2I) diffusion models (DMs) have achieved remarkable generative quality but still exhibit risk of producing biased and inappropriate images. A promising line of prior work aims to mitigate this issue by learning interpretable, linearly controllable concepts from semantic spaces, such as the U-Net bottleneck; however, these methods rely entirely on U-Net architectures and cannot be generalized to modern ViT-based DMs, including FLUX and PixArt. In this work, we present an architecture-agnostic framework for discovering interpretable and linearly controllable semantic attributes across any T2I DM backbone. Theoretically, we show that multi-modal attention heads in ViT-based DMs exhibit a linear semantic structure: injected concept vectors combine linearly at the attention level and induce near-linear effects at the model output, satisfying homogeneity and additivity. These theoretical results are aligned with and supported by empirical experiments. Building on this insight, we introduce a method that learns external concept vectors, which are added to the multi-modal attention heads for ViT-based DMs or to the bottleneck layer for U-Net-based DMs, while keeping pretrained models frozen. Experiments across SDXL, SD3.5, PixArt, and FLUX demonstrate that these concept vectors provide interpretability, linearity, and significantly improved fairness while preserving visual fidelity. The code and demo are available at https://github.com/Moslem-Sh21/responsible-t2i-diffusion.

Modern AI systems can generate realistic images from text descriptions, but they often reflect harmful biases found in their training data. For example, when asked to create images of certain professions, these systems may disproportionately depict specific genders or racial groups. This raises concerns about fairness, safety, and responsible use. In this work, we introduce a simple and general method to make image generation more fair and controllable without retraining the entire model. Our key idea is to learn small, interpretable “adjustment vectors” that represent specific visual concepts, such as gender, age, or safety-related attributes. These vectors can be added to an existing image generator to guide its outputs in a predictable way. We show that these concepts can be combined and scaled smoothly, allowing users to adjust multiple attributes independently. For example, the system can generate a balanced set of images across different genders and races for the same profession, while still preserving image quality and realism. Our method works across a wide range of modern image generation models. Overall, this approach provides a transparent and flexible way to improve fairness and safety in AI-generated images. It can help developers better understand and control how these systems behave, supporting more responsible deployment in real-world applications.