When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents

Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under adversarial settings while preserving or even improving task success rate in benign environments.

Computer-use agents are becoming better at completing digital tasks for people, such as browsing websites, using apps, and editing files. However, these agents can still go off-task: they may follow misleading instructions hidden in a webpage, delete or change files the user did not ask them to touch, or interact with unrelated applications. We study how to detect these misaligned actions before they happen. We create MisActBench, a collection of realistic computer-use trajectories in which humans label whether each agent action follows the user’s intent or goes off-task. We also build DeAction, a runtime guardrail that checks each proposed action, explains why a suspicious action may be wrong, and asks the agent to revise its plan instead of simply stopping it. In experiments, DeAction detects many misaligned actions, greatly reduces successful attempts to mislead agents when malicious content is present, and keeps normal task performance largely intact. This work is a step toward computer-use agents that follow users’ intentions more faithfully and reliably.