In-Context Generation with Regional Constraints for Instructional Video Editing

The In-context generation paradigm has demonstrated strong power in instructional image editing for better synthesis quality. Nevertheless, shaping such in-context learning for instructional video editing is not trivial. Without specifying editing regions, the results can suffer from the issue of inaccurate editing regions and the token interference between different areas. To address these, we present ReCo, a new instructional video editing paradigm that novelly delves into Regional Constraint modeling between editing and non-editing areas. Technically, ReCo width-wise concatenates source and target video for joint denoising. In model training, ReCo formulates regional constraints with two regularization terms, i.e., latent and attention regularization, on one-step backward denoised latents and attention maps, respectively. The former increases the latent discrepancy of the editing region between source and target videos while reducing that of non-editing areas, emphasizing editing area modification and alleviating unexpected content generation. The latter suppresses the attention of tokens in the editing region to the tokens in counterpart of the source video, thereby mitigating their interference during novel object generation in target video. Furthermore, we propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs. Extensive experiments conducted on four major instruction-based video editing tasks verify the superiority of ReCo. Code is available at https://github.com/HiDream-ai/ReCo.

Existing instruction-based video editing models are often limited by the lack of large-scale, high-quality training data and inaccurate editing regions, which hinder both research progress and practical usability. To address these two challenges, we first build ReCo-Data, a large video editing dataset with 500K instruction-video pairs covering object addition, removal, replacement, and style transfer. Human evaluation shows that 91.6% of its samples are high-quality. Then, inspired by the success of in-context learning in image editing, we explore its adaptation to instructional video editing. However, this adaptation is not straightforward, as video editing often suffers from inaccurate editing regions and interference between different areas. To address these issues, we present ReCo, a new instructional video editing paradigm that introduces regional constraint modeling between editing and non-editing areas to achieve accurate editing-region localization and reduce token interference between different regions. Experiments show that ReCo follows editing instructions more accurately, produces more natural edited videos, and better preserves unedited regions compared with existing methods.