Poster
in
Workshop: The Second Workshop on Agents in the Wild: Safety, Security, and Beyond

SafeClawBench: An Operating-System Perspective on Evaluating the Security of Claw-like Agent Systems

Peizhi Niu ⋅ Shangding Gu ⋅ Wenjie Qu ⋅ Tianneng Shi ⋅ Yuankai Li ⋅ Ahmad Al-Tawaha ⋅ Hend Alzahrani ⋅ Vincent Siu ⋅ Boyi Li ⋅ Chenguang Wang ⋅ Jiaheng Zhang ⋅ Basel Alomair ⋅ Ming Jin ⋅ Muhao Chen ⋅ Chi Wang ⋅ Costas J. Spanos ⋅ Dawn Song

Project Page [ OpenReview]

Abstract

Claw-like AI agents (e.g., OpenClaw) are always-on processes running inside the user's environment with persistent access to credentials, files, tools, and external services, functioning as a second operating system (OS). However, this depth of access, together with their widespread adoption, makes the consequences of any security failure far more severe than those of other agents. Despite this, OpenClaw lacks an effective security evaluation method: most existing evaluation benchmarks focus on the security of the model's responses and tool calls, but overlook the OS nature of Claw-like AI agents, thus underestimating their security risks. To address this gap, we measure the security of Claw-like AI agents through the lens of OS that asks whether each of agents' components preserves its corresponding OS-style security invariants. Building on this perspective, we introduce SafeClawBench: a benchmark of 436 adversarial tasks organized along four invariant-aligned dimensions. Each task was executed inside a containerized replica of a real agent platform with canary-marked credentials and was evaluated by automated taint tracking across nine output channels. We evaluated three platforms (OpenClaw, NemoClaw, and SecLaw) and five frontier LLMs. The highest overall attack success rate reaches 69%, and malicious plugins succeed in 100% regardless of the underlying LLM since they are unhardened. Moreover, although platform-level hardening is effective, its efficacy varies between different LLM: SecLaw (a streamlined variant of OpenClaw with added security defenses) cuts GPT-5.4's attack success rate from 69% to 22%, while Claude-Opus-4.6 already sits near a 20% security floor on every platform and gains almost nothing from hardening. The results expose the inadequacy of current defenses and point to possible directions for the future defense design of Claw-like agents. Code and data are available at https://anonymous.4open.science/r/SafeClawBench-C948.

Chat is not available.