DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions

GUI agents - vision-based models that control desktops, web browsers, and mobile devices through their graphical interfaces - promise to automate a wide range of digital tasks. While million-scale datasets have enabled substantial progress on click-grounding, drag-grounding data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. We introduce DragOn, a drag-grounding benchmark and training dataset covering four domains: text highlighting, cell selection, element resizing and slider manipulation. The data is comprised of 286K training screenshots and 3.5M training tasks, plus a 2{,}000-example held-out evaluation suite. We evaluate proprietary (GPT, Claude, Gemini) and open-source (Qwen, Kimi, Holo3) models, as well as a VLM fine-tuned on our training data. Results suggest that our dataset could improve performance of state-of-the-art models on downstream computer-use tasks.