Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning

Instruction-based image editing (IIE) has advanced rapidly with the success of diffusion models. However, existing efforts primarily focus on simple and explicit instructions to execute editing operations such as adding, deleting, moving, or swapping objects. They struggle to handle more complex implicit hypothetical instructions that require deeper reasoning to infer plausible visual changes and user intent. Additionally, current datasets provide limited support for training and evaluating reasoning-aware editing capabilities. Architecturally, these methods also lack mechanisms for fine-grained detail extraction that support such reasoning. To address these limitations, we propose Reason50K, a large-scale dataset specifically curated for training and evaluating hypothetical instruction–reasoning image editing, along with ReasonBrain, a novel framework designed to reason over and execute implicit hypothetical instructions across diverse scenarios. Reason50K includes over 50K samples spanning four key reasoning scenarios: Physical, Temporal, Causal, and Story reasoning. ReasonBrain leverages Multimodal Large Language Models (MLLMs) for editing guidance generation and a diffusion model for image synthesis, incorporating a Fine-grained Reasoning Cue Extraction (FRCE) module to capture detailed visual and textual semantics essential for supporting instruction reasoning. To mitigate the semantic loss, we further introduce a Cross-Modal Enhancer (CME) that enables rich interactions between the fine-grained cues and MLLM-derived features. Extensive experiments demonstrate that ReasonBrain consistently outperforms state-of-the-art baselines on reasoning scenarios while exhibiting strong zero-shot generalization to conventional IIE tasks. Our dataset and code will be released publicly.

In practice, users may not always specify an explicit editing operation, but instead express their intent through hypothetical scenarios, such as asking what a scene would look like after time passes. Existing instruction-based image editing methods mainly handle explicit instructions, such as adding, deleting, moving, or swapping objects, and struggle to infer plausible visual changes from such implicit hypothetical instructions. We address this by introducing Reason50K, a dataset of over 50K samples covering physical, temporal, causal, and story reasoning scenarios. We also propose ReasonBrain, a framework that uses multimodal language models to infer editing guidance and diffusion models to generate edited images, while extracting fine-grained visual and textual cues to inject richer reasoning details. Experiments show that ReasonBrain consistently outperforms existing image-editing methods on reasoning scenarios and also generalizes well to conventional instruction-based image editing tasks.