On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning

Semi-supervised imitation learning (SSIL) consists in learning a policy from a small dataset of action-labeled trajectories and a much larger dataset of action-free trajectories. Some SSIL methods learn an inverse dynamics model (IDM) to predict the action from the current state and the next state. An IDM can act as a policy when paired with a video model (VM-IDM) or as a label generator to perform behavior cloning on action-free data (IDM labeling). In this work, we first show that VM-IDM and IDM labeling learn the same policy in a limit case, which we call the IDM-based policy. We then argue that the previously observed advantage of IDM-based policies over behavior cloning is due to the superior sample efficiency of IDM learning, which we attribute to two causes: (i) the ground-truth IDM tends to be contained in a lower complexity hypothesis class relative to the expert policy, and (ii) the ground-truth IDM is often less stochastic than the expert policy. We argue these claims based on insights from statistical learning theory and novel experiments, including a study of IDM-based policies using recent architectures for unified video-action prediction (UVA). Motivated by these insights, we finally propose an improved version of the existing LAPO algorithm for latent action policy learning. We experiment on the Procgen, Push-T and LIBERO benchmarks.

We teach robots how to accomplish complex tasks by asking human operators to collect example demonstrations by controlling the robot. Teaching the robot these skills requires a vast amount of these demonstrations, and collecting datasets that are large enough is labor-intensive and costly. We investigate methods that instead learn from widely available video data, like videos on YouTube, and a comparatively small amount of human demonstrations. These methods often learn by looking at consecutive video frames and calculating what action caused the scene to change. This is known as learning the "inverse dynamics." In this work, we show that two of these seemingly different methods are actually doing the exact same thing under the hood, which allows us to study them together. We then investigate particular factors explaining why learning inverse dynamics is an "easier" learning problem which will require less data. We use our insights to suggest improvements to existing algorithms, allowing them to learn skills from even less data than before.