MiVE: Multiscale Vision-language features for reference-guided video Editing

Reference-guided video editing takes a source video, a text instruction, and a reference image as inputs, requiring the model to faithfully apply the instructed edits while preserving original motion and unedited content. Existing methods fall into two paradigms, each with inherent limitations: decoupled encoders suffer from modality gaps when processing instructions and visual content independently, while unified vision-language encoders lose fine-grained spatial details by relying solely on final-layer representations. We observe that VLM layers encode complementary information hierarchically—early layers capture localized spatial details essential for precise editing, while deeper layers encode global semantics for instruction comprehension. Building on this insight, we present \textbf{MiVE} (\textbf{M}ult\textbf{i}scale \textbf{V}ision-language features for reference-guided video \textbf{E}diting), a framework that repurposes VLMs as multiscale feature extractors. MiVE extracts hierarchical features from Qwen3-VL and integrates them into a unified self-attention Diffusion Transformer, eliminating the modality mismatch inherent in cross-attention designs. Experiments demonstrate that MiVE achieves state-of-the-art performance by ranking highest in human preference, outperforming both academic methods and commercial systems.

People often want to tweak a video—change someone's hair color, swap a background, or relight a scene—without disturbing the rest. Today's AI editing tools struggle: some rely on two separate "brains," one to read the request and one to look at the video, that fail to coordinate; others use a single combined brain but only listen to its final answer, missing the precise shapes and outlines it noticed earlier on. We found that an AI that sees and reads together works a bit like a person who first scans a photo to register exact outlines, and only afterward forms a deeper interpretation. Our system, MiVE, listens at both stages at once and feeds both into the part of the model that paints the new video, treating the request, the example image, and the original footage as one connected scene instead of three disconnected pieces. When viewers compared edited videos side by side, they preferred MiVE's results over leading research systems and well-known commercial tools, making creative video editing more reliable and accessible to non-experts.