Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos. Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, we construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5–20\% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.

When you click buttons or type into apps — editing a photo, booking a flight, filing a form — you adapt effortlessly across thousands of different software interfaces. Researchers are racing to build AI assistants that can do the same: look at a screen and operate any program on your behalf. The biggest obstacle is data. Today's AI assistants learn from small, hand-labeled collections of step-by-step examples, which restricts them to a narrow set of apps and tasks. Yet the internet already overflows with people demonstrating exactly this: tutorial videos that walk through how to use software, screen by screen. We built Video2GUI, an automated pipeline that converts these tutorial videos into training material. It first sifts through 500 million internet videos to identify genuine, high-quality software walkthroughs, then watches each one to recognize every action — what was clicked, typed, or dragged, and exactly where on the screen. The result is WildGUI, the largest open dataset of its kind: 12 million recorded interactions across more than 1,500 real-world apps and websites. When existing AI models are trained on WildGUI, their ability to operate unfamiliar software improves by 5–20%, matching or surpassing the best systems available. This brings general-purpose digital assistants noticeably closer to working with any app, anywhere.