UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning

Recent advances in large-scale diffusion models have intensified concerns about their potential misuse, particularly in generating realistic yet harmful or socially disruptive content. This challenge has spurred growing interest in effective machine unlearning, the process of selectively removing specific knowledge or concepts from a model without compromising its overall generative capabilities. Among various approaches, Low-Rank Adaptation (LoRA) has emerged as an effective and efficient method for fine-tuning models toward targeted unlearning. However, LoRA-based methods often exhibit limited adaptability to concept semantics and struggle to balance removing closely related concepts with maintaining generalization across broader meanings. Moreover, these methods face scalability challenges when multiple concepts must be erased simultaneously. To address these limitations, we introduce UnHype, a framework that incorporates hypernetworks into single- and multi-concept LoRA training. The proposed architecture can be directly plugged into Stable Diffusion as well as modern flow-based text-to-image models, where it demonstrates stable training behavior and effective concept control. During inference, the hypernetwork dynamically generates adaptive LoRA weights based on the CLIP embedding, enabling more context-aware, scalable unlearning. We evaluate UnHype across several challenging tasks, including object erasure, celebrity erasure, and explicit content removal, demonstrating its effectiveness and versatility.

Recent advances in generative AI models have intensified concerns about their potential misuse, particularly in generating realistic yet harmful or socially disruptive content. In the paper, we present simple yet effective method for making the image diffusion models forget certain concepts. To this end, we use a minimal intrusion to the model architecture by inserting a matrix in crucial components, without modifying the weights. We use a mechanism called hyper-networks -- neural networks that are capable of producing other neural networks. To evaluate efficiency of our idea, we perform a number of experiments: one of them is to ensure that the diffusion model trained on the internet data does not produce pornographic content. Other tasks include forgetting a given number of celebrity identities, for example, preventing the diffusion model from generating images depicting Bruce Willis. Those benchmarks demonstrate the usefulness of our approach in the real-world scenarios, where protecting privacy and mitigating deep-fake is crucial. From a technical perspective, our method achieved state-of-the-art performance in effectively erasing unwanted concepts while also enabling scalability to a substantially larger number of concepts to be forgotten. To facilitate future research and enable others to build upon our work, we release fully reproducible code together with all configuration files.