Counterfactual Trace Auditing of LLM Agent Skills

Large Language Model agents are increasingly augmented with agent skills: structured Markdown documents that inject domain procedural knowledge at run time. Skill marketplaces and personal skill libraries are growing rapidly, but current evaluation methods remain limited. Most deployed benchmarks report only pass rate before and after a skill is attached, treating the skill as a black-box change to agent behavior. We introduce Counterfactual Trace Auditing (CTA), a framework for measuring how a skill changes agent behavior. CTA pairs each with-skill agent trace with a without-skill counterpart on the same task, segments both traces into goal-directed phases, aligns the phases, and emits structured Skill Influence Pattern (SIP) annotations. These annotations describe the behavioral effect of a skill rather than only its task outcome. We instantiate CTA on SWE-Skills-Bench with Claude across 49 software engineering tasks. The resulting audit reveals a clear evaluation gap. Pass rate changes by only +0.3 percentage points on average, suggesting little aggregate effect. Yet CTA identifies 522 SIP instances across the same paired traces, showing that skills substantially reshape agent behavior even when pass rate is nearly unchanged. The audit also separates several recurring effects that pass rate cannot detect, including literal template copying, off-task artifact creation, excess planning, and task recovery. Three findings emerge. First, high-baseline tasks contain most of the observed skill effects, although their pass rate is already saturated and therefore cannot reflect those effects. Second, tasks with moderate baseline performance show the most recoverable gain, but often at substantially higher token cost. Third, the dominant SIP type varies systematically with baseline performance: surface anchoring is most common on ceiling tasks, concept bleed is most common on mid-range tasks, and excess planning is most common on floor tasks.