ReasonEdit: Editing Vision--Language Models using Human Reasoning

Model editing aims to correct errors in large, pretrained models without altering their unrelated behaviors. While some recent works have edited vision–language models (VLMs), no existing editors tackle reasoning-heavy tasks, which typically require humans and models to reason about images. We therefore propose ReasonEdit, the first VLM editor to let users explain their reasoning during editing, introducing a new, practical model editing setup. ReasonEdit continuously stores human reasoning in a codebook, and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Across four VLMs on multiple rationale-based visual question answering datasets, ReasonEdit achieves state-of-the-art editing performance, ultimately showing that using human reasoning during editing greatly improves edit generalization.

Model editing aims to correct model errors at inference time, without expensive retraining or changing unrelated behaviors. While recent works have edited vision-language models for label correction, no existing editors tackle reasoning-heavy tasks that require detailed reasoning by humans or models. We propose ReasonEdit, the first reasoning-enhanced editor for vision-language models. When a model makes an error, a user provides detailed factual reasoning statements that explain how the image and knowledge lead to the correct answer. ReasonEdit stores these reasoning facts in a codebook and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Building on the observation that vision and language layers can each be biased toward their own modality, we treat multimodal embeddings as nodes in a graph and measure their modularity, a widely used measure of community structure in network science. ReasonEdit then uses a novel dual embedding method to balance multimodal topology when embedding image–text pairs as keys in the codebook, enabling much more accurate retrieval. Across multiple vision-language models and datasets, ReasonEdit achieves state-of-the-art editing performance and shows that human reasoning can greatly improve edit generalization by generalizing edits to unseen samples that share the same underlying reasoning.