AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows

Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent how to perform a domain task. As skill repositories grow, developers need automated, trustworthy quality signals on every change to a skill, yet most evaluation today is anecdotal: a developer asks an agent to "try the skill," watches a demo turn, and forms a subjective impression of whether it works. This produces neither reproducibility across runs nor comparability across versions, and it scales poorly to multi-skill marketplaces where a single regression can silently break dozens of downstream workflows. We present AEVAL (Agentic Evaluation), a CI-integrated framework that replaces this anecdotal practice with a deterministic, reproducible test pipeline for agentic skills. The framework treats every skill change as a triggered test event, runs each skill against a developer-declared evaluation contract (eval.config) inside an automated executor, and emits a structured, evidence-grounded quality signal that downstream CI can route on. A key technical ingredient is a structural separation between the executor and the grader, which prevents a subtle but pervasive failure mode of agentic evaluation: an agent that silently self-corrects during execution and then grades its own patched outputs as passing. Our contributions are: (i) a deterministic, change-triggered evaluation protocol for skills with per-skill evaluation contracts and per-run artifact schemas; (ii) a formalization of self-correction bias as a distinct failure mode of naive agentic evaluators; (iii) a structural executor/grader separation with a first-attempt grading rule and explicit self-correction tracking; and (iv) a tiered, grounded evidence suggestion scheme (LV1 causal fixes, LV2 quality improvements) posted as inline merge-request comments. We validate the framework against a runtime-agnostic interface tested with multiple popular agent SDKs. Across a set of real skills in a production agentic stack, the protocol converts spurious 100% pass rates into reproducible first attempt fail signals with an auditable record of every fix the executor applied, enabling reliable routing and stopping decisions in downstream agentic workflows.