PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

Predicting future scene representations is a crucial task for enabling robots to understand andinteract with the environment. However, most existing methods rely on videos and simulationswith precise action annotations, limiting their ability to leverage the large amount of avail-able unlabeled video data. To address this challenge, we propose PlaySlot, an object-centricvideo prediction model that infers object representations and latent actions from unlabeledvideo sequences. It then uses these representations to forecast future object states and videoframes. PlaySlot allows the generation of multiple possible futures conditioned on latent actions,which can be inferred from video dynamics, provided by a user, or generated by a learned actionpolicy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlotoutperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled videodemonstrations. Videos and code are available on our project website.

Robots need to understand what’s happening around them and guess what might happen next. Most current methods need lots of detailed instructions to learn this, which takes a lot of time and effort. Our method, called PlaySlot, learns just by watching videos---like how people can learn by observing. It figures out what the important objects are, how they move, and then imagines what could happen next. This helps robots learn to do tasks even when we don’t give them step-by-step directions. It’s more flexible, easier to use, and works better than other tools.