Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism

Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors inherent to behavioral cloning (BC), yet its adversarial optimization frequently leads to training instability. A class of non-adversarial Q-based imitation learning (IL) methods, exemplified by IQ-Learn, has emerged to address this instability and is widely believed to outperform BC by leveraging online environment interactions. In this paper, we revisit IQ-Learn and prove that it in fact reduces to BC: it admits an imitation gap lower bound with quadratic dependence on the horizon and therefore remains susceptible to compounding errors. Our theoretical analysis reveals why online interactions fail to help: IQ-Learn uniformly suppresses Q-values for all actions at states not covered by demonstrations, preventing generalization beyond demonstrations. To address this fundamental limitation, we introduce Dual Q-DM, a new Q-based IL method built on Bellman constraints. Crucially, Bellman constraints drive value flow: Q-values propagate from demonstrated to unvisited states through environment dynamics, enabling generalization beyond demonstrations. We prove that Dual Q-DM is equivalent to AIL and can recover expert actions at unvisited states, thereby mitigating compounding errors. To the best of our knowledge, Dual Q-DM is the first non-adversarial IL method that is theoretically guaranteed to eliminate compounding errors. Experimental results further corroborate our theoretical findings.

Teaching a robot to move like an expert — say, learning to walk by watching a skilled athlete — sounds simple: just copy what you see. But small copying mistakes snowball the moment the robot encounters an unfamiliar situation, leading to catastrophic failures. This is known as "compounding errors," and it is one of the central challenges in teaching AI through imitation. A popular approach called IQ-Learn was widely believed to fix this by letting the AI practice in the real environment, not just copy from recordings. We prove, however, that IQ-Learn secretly reduces to simple copying: it learns nothing useful about situations it has never seen before. We identified the missing ingredient: value flow. Just as water flows downhill, value signals should propagate backward from situations the expert visited to nearby situations the expert did not. Our new method implements this idea: when the AI evaluates a never-seen situation, it asks "does this action lead toward places the expert went?" — and answers correctly, even without direct experience there. We prove mathematically that our method is the first to provably eliminate compounding errors without the training instability of adversarial approaches. Experiments on robotic control tasks confirm our findings.