ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision

Modern computer-use agents (CUA) must perceive a screen as a structured state, what elements are visible, where they are, and what text they contain, before they can reliably ground instructions and act. Yet, most available grounding datasets provide sparse supervision, with insufficient and low-diversity labels that annotate only a small subset of task-relevant elements per screen, which limits both coverage and generalization; moreover, practical deployment requires efficiency to enable low-latency, on-device use. We introduce ScreenParse, a large-scale dataset for complete screen parsing, with dense annotations of all visible UI elements (boxes, 55-class types, and text) across 771K web screenshots (21M elements). ScreenParse is generated by Webshot, an automated, scalable pipeline that renders diverse urls, extracts annotations and applies VLM-based relabeling and quality filtering. Using ScreenParse, we train ScreenVLM, a compact, 316M-parameter vision language model (VLM) that decodes a compact ScreenTag markup representation with a structure-aware loss that upweights structure-critical tokens. ScreenVLM substantially outperforms much larger foundation VLMs on dense parsing (e.g., 0.592 vs. 0.294 PageIoU on ScreenParse) and shows strong transfer to public benchmarks. Moreover, finetuning foundation VLMs on ScreenParse consistently improves their grounding performance, suggesting that dense screen supervision provides transferable structural priors for UI understanding. The dataset, model and code will be made publicly available.

Many AI systems are now being built to use computers by looking at the screen and deciding where to click, type, or scroll. For this to work reliably, the system needs to understand the whole screen, not just one button or icon mentioned in an instruction. In this paper, we introduce ScreenParse, a large dataset that describes visible elements on web screens, including where they are, what type of element they are, and what text they contain. We also introduce Webshot, an automated pipeline for creating these annotations at scale, and ScreenVLM, a compact model trained to turn a screenshot into a structured description of the screen. Our experiments show that dense screen-level supervision helps models understand user interfaces more completely and improves performance across different screen understanding and action-selection benchmarks. These results suggest that better screen parsing can be a useful building block for future computer-use agents, accessibility tools, and automated user-interface testing systems.