Adversarial Dual On-Policy Distillation from Expressive Teacher

Learning from demonstrations in embodied control is often cast as behavioral cloning, and recent diffusion or flow-matching policies improve this paradigm by modeling multi-modal expert actions. Yet these methods remain offline supervised learners: the policy is trained only on expert states and receives no corrective signal on the states it actually visits. On-policy distillation (OPD) offers a natural remedy, but standard OPD assumes a strong fixed teacher, which is unavailable in demonstration-only control. We propose FA-OPD, an adversarial dual on-policy distillation method in which a Flow Matching (FM) teacher is learned from demonstrations and co-trained with a lightweight MLP student. The teacher provides two complementary signals on student rollouts. The reward channel learns an expert-likeness objective over state-action pairs and drives online exploration through long-horizon policy optimization. The action channel supplies dense local targets at student-visited states, stabilizing exploitation. FA-OPD couples them so that reward distillation enables generalization beyond point-wise demonstrations, while action distillation keeps exploration anchored near expert-like behavior. Across six robot navigation, manipulation, and locomotion benchmarks, FA-OPD beats strong baselines and shows much stronger robustness under noisy or limited demonstrations. Source code: https://github.com/vanzll/FA-OPD.

Many robots learn skills—like walking, navigating to a goal, or picking up objects—by watching a human expert and copying what they did. This works until the robot makes a small mistake and ends up in a situation the expert never showed it. With no idea how to recover, its errors pile up. The problem gets worse when the demonstrations are few, noisy, or imperfect. We introduce a method, FA-OPD, that fixes this by letting the robot practice on its own while a "teacher" watches and gives feedback. The teacher is trained only from the same expert demonstrations, but instead of just grading the robot once at the end, it coaches in two complementary ways: it tells the robot how expert-like each of its own attempts looks, which encourages it to explore and improve, and it suggests what a good action would have been in the exact situations the robot actually got into, which keeps that exploration grounded in sensible behavior. The teacher is only needed during training; the deployed robot is a small, fast model. Across six robot tasks, FA-OPD outperforms strong existing methods and stays reliable even when the demonstrations are limited or low-quality.