Hierarchical Procedural Meta-Reasoning for Generalizable Multimodal Agents

While multimodal agents can achieve strong performance through fine-tuning, their ability to generalize remains limited in complex real-world tasks such as mobile navigation, where diverse applications, frequent system changes, and customized workflows are common in practice. We argue that a fundamental bottleneck lies in whether an agent possesses sufficient task-specific procedural knowledge to accomplish a given goal. Such procedural knowledge may be provided by the general capabilities of large language models, or obtained from additional external resources such as web search when necessary. Based on this view, we propose Procedure-Aware Multimodal Agent with Meta Reasoning, a framework that explicitly represents task knowledge as natural-language procedures and trains a procedure-aware grounded agent to condition its actions on this knowledge. By learning to leverage procedural knowledge from different sources, our approach enables robust generalization across tasks, applications, interface versions, and multi-app workflows, achieving substantial improvements on challenging Android benchmarks.

Mobile assistants powered by AI can perform tasks on smartphones, such as changing settings or booking tickets. However, these agents often fail when deployed on unfamiliar or updated apps because they lack the task-specific knowledge needed to complete a task. To address this problem, we developed a method that helps AI agents robustly follow explicit step-by-step procedures. As a result, the agents can rely on these procedures to perform unfamiliar tasks more effectively. We also developed a mechanism that allows large language models to generate procedures automatically and selectively seek external information when their existing knowledge is insufficient. We evaluated our approach on challenging Android benchmarks involving unseen apps, app updates, and multi-app tasks. Our method substantially improves task success while avoiding unnecessary information searches. These results suggest that combining explicit procedures with selective information retrieval can make AI assistants more reliable in real-world mobile environments.