Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion

GUI grounding maps natural language instructions to the correct interface elements, serving as the perception foundation for GUI agents. Existing approaches predominantly rely on fine-tuning multimodal large language models (MLLMs) using large-scale GUI datasets to predict target element coordinates, which is data-intensive and generalizes poorly to unseen interfaces. Recent attention-based alternatives exploit localization signals in MLLMs attention mechanisms without task-specific fine-tuning, but suffer from low reliability due to the lack of explicit and complementary spatial anchors in GUI images. To address this limitation, we propose Trifuse, an attention-based grounding framework that explicitly integrates complementary spatial anchors. Trifuse integrates attention, OCR-derived textual cues, and icon-level caption semantics via a Consensus-SinglePeak (CS) fusion strategy that enforces cross-modal agreement while retaining sharp localization peaks. Extensive evaluations on four grounding benchmarks demonstrate that Trifuse achieves strong performance without task-specific fine-tuning, substantially reducing the reliance on expensive annotated data. Moreover, ablation studies reveal that incorporating OCR and caption cues consistently improves attention-based grounding performance across different backbones, highlighting its effectiveness as a general framework for GUI grounding.

Many AI assistants can now interact with apps, websites, and desktop software on your behalf — but to do so reliably, they first need to understand what's on the screen and locate the right button, icon, or text to click. This task, called GUI grounding, is surprisingly difficult: today's best methods typically require large collections of human-annotated screenshots to learn from, and often fail when encountering a new app or an unfamiliar interface layout. We introduce Trifuse, a method that improves this localization ability without needing any additional task-specific training data. The key idea is to combine three complementary sources of information: (1) where the AI model's internal "attention" naturally focuses on the screen — similar to where a person's eyes would gravitate; (2) text detected directly from the screenshot; and (3) semantic descriptions of icons and visual controls. Rather than relying on any single signal, Trifuse fuses all three to make more reliable predictions, especially in cases where one source of information alone would be misleading. Across four standard benchmarks, Trifuse improves the accuracy of existing attention-based approaches by up to 35 percentage points, with no GUI-specific training required. This suggests a promising path toward AI assistants that can generalize more readily to new software environments — potentially benefiting everyday tools like screen readers, automated testing, and personal productivity agents.