Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Abstract
Hand-written skill libraries make computer-using agents easier to inspect, but they are expensive to build. We ask whether skills mined from graphical user interface (GUI) trajectories can replace this manual step. Our pipeline segments action traces, clusters segments into candidate skills, and uses the resulting labels to train Qwen3-8B with Group Relative Policy Optimization (GRPO) from the base model. The clusters are meaningful on the source benchmark: five of eight have at least 0.95 purity against ground-truth InteraSkill Workflows (IW) labels. The trained policy does not show verified broad cross-domain transfer. IW step accuracy rises only from 18.5% to 20.5%, and BrowseComp+ changes from 43.5% to 43.3%. More importantly, trivial frequency baselines remain stronger than several learned components: the most common skill predictor beats the proposed multi-layer perceptron (MLP) and GRPO policies on IW accuracy, and it beats Auto-SKILL.md on normalized edit distance at every evaluated training size. Fresh current-run evaluations on Mind2Web, live WebArena policy execution, and live WorkArena are still missing. The result is mostly negative: trajectory mining can produce readable skill structure, but the current learned methods do not yet beat simple statistical baselines or support a reliable cross-domain policy.