Timezone: »

 
Poster
Multi-View Masked World Models for Visual Robotic Manipulation
Younggyo Seo · Junsu Kim · Stephen James · Kimin Lee · Jinwoo Shin · Pieter Abbeel

Tue Jul 25 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #338
in Poster Session 2 »
Event URL: https://sites.google.com/view/mv-mwm »

Visual robotic manipulation research and applications often use multiple cameras, or views, to better perceive the world. How else can we utilize the richness of multi-view data? In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation. Specifically, we train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints and then learn a world model operating on the representations from the autoencoder. We demonstrate the effectiveness of our method in a range of scenarios, including multi-view control and single-view control with auxiliary cameras for representation learning. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization and transferring the policy to solve real-robot tasks without camera calibration and an adaptation procedure. Video demonstrations are available at: https://sites.google.com/view/mv-mwm.

[ Slides]  [ PDF] 

Author Information

Younggyo Seo (KAIST / UC Berkeley)
Junsu Kim (KAIST)
Stephen James (Dyson)
Kimin Lee (Google)
Jinwoo Shin (KAIST)
Pieter Abbeel (UC Berkeley & Covariant)

More from the Same Authors