LayerT2V: A Unified Multi-Layer Video Generation Framework

Text-to-video generation has advanced rapidly, but existing methods typically output only the final composited video and lack editable layered representations, limiting their use in professional workflows. We propose \textbf{LayerT2V}, a unified multi-layer video generation framework that produces multiple semantically consistent outputs in a single inference pass: the full video, an independent background layer, and multiple foreground RGB layers with corresponding alpha mattes. Our key insight is that recent video generation backbones use high compression in both time and space, enabling us to serialize multiple layer representations along the temporal dimension and jointly model them on a shared generation trajectory. This turns cross-layer consistency into an intrinsic objective, improving semantic alignment and temporal coherence. To mitigate layer ambiguity and conditional leakage, we augment a shared DiT backbone with LayerAdaLN and layer-aware cross-attention modulation. LayerT2V is trained in three stages: alpha mask VAE adaptation, joint multi-layer learning, and multi-foreground extension. We also introduce \textbf{VidLayer}, the first large-scale dataset for multi-layer video generation. Extensive experiments demonstrate that LayerT2V substantially outperforms prior methods in visual fidelity, temporal consistency, and cross-layer coherence. To facilitate future research, we will release the code and dataset upon publication.

AI can now create realistic videos from a text description, but it gives you one finished, flattened clip, much like a painting where every element is fused onto a single canvas. Professional editors instead work in separate layers, keeping the subject, the background, and the cutout mask apart so they can swap a backdrop or adjust a character without redoing the whole scene. Existing AI video tools do not provide these layers, which makes such edits difficult. We built LayerT2V, a system that generates a video together with all of its editable layers at once, including the full scene, a clean background, the foreground subjects, and the transparency masks that cut them out. Our key idea was to arrange the layers one after another in time and let the AI create them together along a single path, so they stay perfectly aligned and consistent with one another. We also built VidLayer, the first large collection of videos pre-separated into these layers, which we used to train the system. This lets creators replace backgrounds, edit a character, or add effects to a single element without regenerating the entire video. By making layered, professional grade editing more accessible, our work could speed up filmmaking, advertising, and education, and we are releasing our code and dataset so that others can build on it.