The Perception Gap: A Survey of VLM Robustness Defenses for Computer-Use Agent Safety

Computer-use agents (CUAs) are systems that perceive screen state and execute GUI and OS actions in live desktop or browser environments; Claude Computer Use, OpenAI Operator, and UI-TARS are already in production. The safety problem is one of visual grounding. Untrusted screen content reaches the agent's reasoning pipeline through a perception layer that is both the primary attack surface and, in existing surveys, the least defended component. Prior CUA safety work covers it as one category among many and draws defenses entirely from systems security, leaving the VLM robustness literature (certified defenses, encoder hardening, safety fine-tuning) largely unengaged. This paper surveys the intersection. We organize seven perception-layer attack families and map nine published VLM robustness defenses against them; we also cover three emerging defense families (access control, formal runtime governance, industrial mechanisms) addressing the action and trajectory layers. Two findings structure the analysis: no published defense fully addresses typographic exploitation or GUI deception in a CUA setting, and VLM robustness defenses degrade substantially in agentic deployment due to distribution and threat-model mismatches that are systematic but addressable. We identify six open problems and the research agenda needed to close them.