Sparse Relaxed-Lasso Steering: Automatic Sparse Autoencoder Feature Selection for Precise Image Editing

Precise, training-free image editing with text-to-image diffusion models requires balancing alignment (faithful realization of the target attribute), consistency (preserving non-target content), and quality (maintaining sharp, artifact-free textures). Sparse autoencoder (SAE) steering offers interpretable, smooth ``slider-like'' control by manipulating SAE feature activations derived from the text encoder; however, existing approaches rely on heuristic feature selection and manual steering-strength tuning, leading to suboptimal trade-offs among the three objectives. We propose Sparse Relaxed-Lasso Steering (SRLS), which casts steering-vector discovery as a convex sparse recovery problem. Exploiting the affine structure of the SAE decoder, SRLS automatically identifies sparse, generalizable support sets via a Lasso objective and then debiases the coefficients using support-restricted ridge refitting. We further replace manual strength tuning with a fixed-budget Bayesian optimization procedure. Across diverse attributes and subjects, SRLS improves the alignment--consistency--quality trade-off over competing methods.

Text-to-image models can generate realistic images, but precise editing remains difficult: adding one attribute may unintentionally change the subject, background, or image quality. We propose Sparse Relaxed-Lasso Steering, a method that automatically finds a small set of internal features related to the desired edit and selects an appropriate editing strength. This lets the model add attributes such as “sunglasses” or “made of stone” while better preserving unrelated parts of the original image. Our results show that this approach makes image editing more accurate, consistent, and controllable without retraining the model.