GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding

Multimodal large language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight -- a model trained for image-grounded reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8\% accuracy, surpassing V2P-7B (50.6\% with 9.6M training samples) and GTA-1-7B (50.1\% with 1.56M training samples).

When AI assistants try to operate a computer or phone for us — clicking a button, opening a menu, filling a form — they often struggle to pinpoint exactly where on the screen to click. On busy, high-resolution displays, even today's best systems miss small targets about half the time, which is a major reason "do this task for me" assistants still feel unreliable. We built GUI-Spotlight, a model that tackles this by mimicking how a person searches a cluttered screen: instead of trying to spot a tiny icon in one glance, it repeatedly zooms in on smaller and smaller regions until the target is in clear view. It does this by learning, on its own, when to crop a region, when to split the screen into quarters, and when to search by color. Trained on roughly 18,500 examples — hundreds of times fewer than competing systems use — GUI-Spotlight matches or beats much larger models on standard screen-localization tests, pointing toward more dependable and far less data-hungry AI assistants for everyday tasks, accessibility tools, and software testing.