DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation

Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models.

Recent foundation models have greatly improved audio-video generation. However, current methods usually handle human-centered tasks separately, such as generating videos from reference identities and voices, editing existing videos, or animating videos with driving audio. They also struggle to control multiple people and voices in the same scene without mixing up identities or speakers. We propose DreamID-Omni, a unified framework for controllable human-centered audio-video generation. It combines different conditions, including reference images, voice timbres, source videos, and driving audio, within one model. To better match each person with the correct voice and attributes, we introduce mechanisms for stronger identity-voice binding and clearer subject-level descriptions. Experiments show that DreamID-Omni achieves strong performance in video quality, audio quality, and audio-visual consistency.