WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces. This paper identifies a critical reliability gap: state-of-the-art agents face distinct robustness challenges in real-world desktop environments characterized by multi-window stacking, occlusion, and visual clutter. To address this, we introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness. Unlike static datasets, our framework parametrically generates complex desktop scenarios by controlling window occlusion, layout density, and semantic similarity, thereby simulating the distribution shifts of authentic workflows. We construct a diverse meta-dataset of 1,356 high-fidelity instruction-target pairs and conduct comprehensive evaluations of five leading MLLMs. Our results demonstrate that while top-tier agents excel in simplified settings, their accuracy declines under partial occlusion. WinDeskGround provides a valuable benchmark to facilitate the assessment and advancement of GUI agent robustness in realistic environments.

Many AI systems today can understand what is on a computer screen and help automate everyday tasks. However, these systems are usually tested in simple, clean setups that do not reflect how people actually use computers. In real life, users often work with many overlapping windows, cluttered layouts, and visually similar elements, which can confuse AI and lead to mistakes. To study this problem, we introduce WinDeskGround, a new testing framework that creates realistic desktop scenarios. Instead of relying on fixed examples, our approach can generate a wide range of situations by controlling how much windows overlap, how crowded the screen is, and how similar different elements look. We also built a dataset of over a thousand examples and used it to evaluate several leading AI systems. Our results show that while these systems perform well in simple environments, their accuracy drops when parts of the screen are hidden or cluttered. This work highlights an important gap between current AI performance and real-world use, and provides a tool to help build more reliable and practical computer assistants.