VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.

Modern AI assistants are beginning to control smartphone apps by looking at the screen and tapping buttons, but it is still unclear whether they can handle the messy situations people face in everyday phone use. Existing tests often focus on narrow, app-specific tasks, so they may make these systems look more capable than they really are. We introduce VenusBench-Mobile, a new test suite built around realistic user needs, such as handling changing app screens, remembering earlier information, comparing options, and completing multi-step tasks. We use it to test a wide range of mobile AI assistants and to understand not only whether they succeed, but also why they fail. Our results show that today’s systems struggle much more on these realistic tasks than on previous tests, especially when they need to correctly understand what is on the screen or remember information across steps. We also find that even strong systems can be fragile when small details in the phone environment change. By revealing these weaknesses, VenusBench-Mobile can help researchers build mobile AI assistants that are more reliable and useful in real-world use.