AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: \url{https://AgentHijack.github.io}.

Autonomous computer use agents powered by large AI models are emerging as capable assistants for completing complex digital workflows — clicking buttons, filling forms, and navigating apps on your behalf. But there is a catch: real-world execution environments are far from ideal. Pop-ups, resolution changes, and competing applications frequently interfere with what the agent sees and does. Today's evaluations rarely test for this, so we do not really know how fragile these agents are. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions — nine everyday disturbances such as pop-ups, accidental clicks, and network errors. Testing today's leading agents, we found that even minor instances of corruption can result in substantial performance degradation: agents repeatedly click buttons that no longer exist, or mistake an accidental keystroke for their own action. To help, we built a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking — a companion that watches the screen and warns when something has gone wrong. This makes agents noticeably more reliable across all corruption types, pushing computer-use AI closer to trustworthy real-world deployment.