When Does Predictive Inverse Dynamics Outperform Behavior Cloning?

Behavior cloning (BC) is a practical offline imitation learning method, but it often fails when expert demonstrations are limited. Recent works have introduced a class of architectures named predictive inverse dynamics models (PIDMs) that combine a future-state predictor with an inverse dynamics model. While PIDMs often outperform BC, the reasons behind their benefits remain unclear. In this paper, we provide a theoretical explanation: PIDMs introduce a tradeoff. Conditioning the IDM on the predicted future state can significantly reduce variance, but the prediction itself introduces additional bias and variance. We establish conditions for PIDMs to achieve higher sample efficiency and lower prediction error than BC, with the gap widening when additional data sources are available. We validate the theoretical insights empirically in 2D navigation tasks, where BC requires up to five times (three times on average) more demonstrations than PIDM to reach comparable performance. Results are also illustrated in a complex 3D environment in a modern video game with high-dimensional visual inputs and stochastic transitions, where BC requires over 66% more samples than PIDM.

Teaching machines to imitate human behavior, like driving a car or playing a game, usually requires many examples of an expert performing the task. Collecting these examples is expensive, so we want methods that learn from as few as possible. The simplest approach, behavior cloning, directly mimics what the expert did in each situation. A newer approach, called PIDM, instead first predicts where the agent should be a moment from now, then figures out which action would get it there. Recent studies show PIDM often needs far fewer demonstrations, but no one had clearly explained why. Our work provides that explanation. Predicting a future situation introduces some error, but considering the predicted future situation helps to identify which action would be effective, reducing uncertainty about the action prediction. In most practical cases, this reduction in uncertainty more than compensates for the prediction error. We identify when and why PIDM is guaranteed to be more efficient than behavior cloning. In experiments, behavior cloning needed up to five times more demonstrations in 2D navigation tasks, and two thirds more in a complex 3D video game, to match PIDM's performance.