SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

Repurposing large vision-language models (LVLMs) as computer use agents (CUAs) has led to substantial breakthroughs, primarily driven by human-labeled data. However, these models often struggle with novel and specialized software, particularly in scenarios lacking human annotations. To address this challenge, we propose SEAgent, an agentic self-evolving framework enabling CUAs to autonomously evolve through interactions with unfamiliar software. Specifically, SEAgent empowers computer-use agents to autonomously master novel software environments via experiential learning, where agents explore new software, learn through iterative trial-and-error, and progressively tackle auto-generated tasks organized from simple to complex. To achieve this goal, we design a World State Model for step-wise trajectory assessment, along with a Curriculum Generator that generates increasingly diverse and challenging tasks. The agent's policy is updated through experiential learning, comprised of adversarial imitation of failure actions and Group Relative Policy Optimization (GRPO) on successful ones. Furthermore, we introduce a specialist-to-generalist training strategy that integrates individual experiential insights from specialist agents, facilitating the development of a stronger generalist CUA capable of continuous autonomous evolution. This unified agent ultimately achieves performance surpassing ensembles of individual specialist agents on their specialized software. We validate the effectiveness of SEAgent across five professional software of OSWorld, ScienceBoard and AndroidWorld. Our approach achieves a significant improvement over a competitive open-source CUA, UI-TARS. All the code and models will be made publicly available to foster further research.

Modern AI assistants can already look at a computer screen and use the mouse and keyboard to do tasks for us. The catch is that they tend to work well only on familiar applications, because someone first had to record lots of step-by-step demonstrations by hand. Whenever a new app appears or an old one is updated, these assistants quickly fall behind. We build SEAgent, a computer-using assistant that teaches itself how to operate unfamiliar software, without any human-recorded tutorials. Much like a curious new user, it explores the software on its own: it tries actions, watches how the screen changes after each click, automatically judges whether the step looked helpful or harmful, and then sets itself increasingly difficult practice tasks. By learning from both its mistakes and its successes, the assistant steadily becomes more capable on its own. Across a range of professional applications — code editors, image and video editors, scientific tools, and mobile apps — SEAgent ends up substantially better than a strong existing open-source assistant. We also show how several "specialist" assistants, each good at one app, can be combined into a single "generalist" that handles many applications well. We hope this is a step toward AI helpers that can adapt to any software on their own, without waiting for humans to label new training data.