Timezone: »

 
Poster
Compositional Video Synthesis with Action Graphs
Amir Bar · Roi Herzig · Xiaolong Wang · Anna Rohrbach · Gal Chechik · Trevor Darrell · Amir Globerson

Tue Jul 20 09:00 PM -- 11:00 PM (PDT) @
in Poster Session 2 »

Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new "Action Graph To Video" synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on CATER and Something-Something V2 datasets, which results in videos that have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions.

Keywords: Computer Vision

Author Information

Amir Bar (Tel Aviv University)
Roi Herzig (Tel Aviv University)
Xiaolong Wang (UCSD)
Xiaolong Wang

Our group has a broad interest around the directions of Computer Vision, Machine Learning and Robotics. Our focus is on learning 3D and dynamics representations through videos and physical robotic interaction data. We explore various means of supervision signals from the data itself, language, and common sense knowledge. We leverage these comprehensive representations to facilitate the learning of robot skills, with the goal of generalizing the robot to interact effectively with a wide range of objects and environments in the real physical world. Please check out our individual research topic of Self-Supervised Learning, Video Understanding, Common Sense Reasoning, RL and Robotics, 3D Interaction, Dexterous Hand.

Anna Rohrbach (UC Berkeley)
Gal Chechik (NVIDIA / Bar-Ilan University)
Trevor Darrell (University of California at Berkeley)
Amir Globerson (Tel Aviv University, Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors