RepoGuardBench: Repository-Borne Prompt Injection Attacks and Lightweight Defenses for Local Coding Agents
Abstract
Repository artifacts such as READMEs, issue bodies, code comments, test logs, and agent-rule files are both program context and potential instruction channels. We introduce RepoGuardBench, which jointly measures repair utility and prompt-injection robustness for local open-weight coding agents. The benchmark contains 80 synthetic tasks with single-source-file fixes and 14 hand-curated multi-file Python repair tasks, five carrier types, and five inert attack goals. In a 2,340-run fractional local grid spanning four Qwen2.5-Coder sizes and eight lightweight defenses, we use matched aggregation to evaluate safety–utility tradeoffs; we also report smaller cross-family local pilots and a separated non-local Claude reference. Code comments and rule files are the highest-risk carriers. Across Qwen2.5-Coder, matched proposed-attempt rates rise from 0% at 1.5B to 22%, 58%, and 83% at 3B, 7B, and 14B, with landed unsafe actions increasing in parallel. Strict-sandbox completion is low but nonzero (2.1%) and is dominated by rule-file-induced test deletion, exposing a semantic gap in path-based action gates. Context-side and action-side defenses show different failure modes, with no statistically clear utility winner among the five primary defenses (D0–D3, D5); defense-prompt wording effects remain exploratory. RepoGuardBench provides a reproducible framework for studying coding-agent security under repository-context interactions.