Beyond Policy Training: Recursive Solution Search from Unannotated Videos

Many real-world tasks are recorded as large collections of unannotated task executions, such as videos, which contain rich information about task progress but lack the supervision assumed by standard reinforcement learning (RL) pipelines. In many practical settings, the goal is not to train a reusable policy but simply to recover one feasible solution, making policy-centered learning unnecessarily costly. We propose Policy-Free Recursive Search (PFR-Search), a framework that directly recovers solutions from unannotated task executions without policy-grounded supervision or policy training. PFR-Search organizes videos into high-level task flows and performs recursive backward-forward search to recover solutions without policy modeling. To evaluate the efficiency of policy-free search in exploiting unannotated data, we use RL as an evaluation interface, incorporating task-flow-aligned intrinsic rewards and compare against video-driven RL methods. Experiments on long-horizon Minecraft tasks show that PFR-Search recovers feasible solutions from unannotated videos with minimal exploration.

Many real-world tasks are recorded in videos, but these videos usually do not include the action labels or reward signals that standard AI training methods need. In many situations, we may not need to train an AI system that can repeatedly solve a task; we may only need to find one feasible way to complete it. This paper studies how to recover such a solution directly from videos. Our method organizes videos into high-level task stages, reasons backward from the goal to infer a roadmap, and then searches forward from the start by following that roadmap. This breaks a long task into smaller steps without first training a full control policy. Experiments on long-horizon Minecraft tasks show that task structure extracted from videos can help recover feasible solutions with limited exploration.