Timezone: »

Compositional Video Synthesis with Action Graphs
Amir Bar · Roi Herzig · Xiaolong Wang · Anna Rohrbach · Gal Chechik · Trevor Darrell · Amir Globerson

Tue Jul 20 07:35 PM -- 07:40 PM (PDT) @ None

Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new "Action Graph To Video" synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on CATER and Something-Something V2 datasets, which results in videos that have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions.

Author Information

Amir Bar (Tel Aviv University)
Roei Herzig (Tel Aviv University)
Xiaolong Wang (UCSD)
Anna Rohrbach (UC Berkeley)
Gal Chechik (NVIDIA / Bar-Ilan University)
Prof. Darrell (University of California at Berkeley)
Amir Globerson (Tel Aviv University, Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors