MidSteer: Optimal Affine Framework for Steering Generative Models

Steering intermediate representations has emerged as a powerful strategy for controlling generative models. However, despite its empirical success, it currently lacks a comprehensive theoretical framework. In this paper, we bridge this gap by formalizing the theory of concept steering. First, we establish a link between steering and affine concept erasure, proving that the standard approach for removing unwanted behaviors is a special case of LEACE (a closed-form method for affine erasure). Next, we formulate a principled theoretical framework for concept switching, LEACE-Switch, and characterize the assumptions under which it provides an optimal affine solution. Building on this analysis, we then introduce MidSteer (Minimal Disturbance concept Steering), a more general affine framework for concept manipulation that relaxes these assumptions and enables directed, minimal-disturbance transformations. We empirically demonstrate that MidSteer performs favorably across a range of tasks, modalities, and architectures, including vision diffusion models and large language models.

A popular way to control modern generative AI systems is called activation steering. Instead of retraining a model, it modifies its internal representations during generation of text or images. This technique has been useful in practice, for example for reducing toxic text, changing visual attributes, or encouraging more desirable behavior. However, many steering methods are still based on simple geometric heuristics. While they can be effective, they often accidentally change unrelated aspects of the output, and it is unclear what objective they are optimizing. In this work, we ask whether activation steering can be put on a stronger mathematical foundation. We build on recent theory of concept erasure, which studies how to remove information about a concept from a model’s internal representations. Our goal is to move beyond erasure and study concept switching: changing one concept into another, such as toxicity into helpfulness or violence into peace, while preserving the rest of the generation. We derive MidSteer, a closed-form method for minimal-disturbance concept switching. Experiments on language and image models show that MidSteer gives more reliable switching and fewer side effects than standard steering. Overall, the work extends concept-erasure theory into a broader theory of controllable concept switching for safer AI generation.