Evaluate activation and output quality

Activation tests answer “did the skill load?” Output evaluations answer “did it help?” You need both.

Create three realistic repository scenarios: a clean release with complete documentation, a dirty repository with a failing test, and a project missing a release requirement. Write the expected decision and the evidence each report must contain before running the agent.

Run every scenario with the skill. Then run the same prompts without it. Compare correctness, evidence, unnecessary actions, missed blockers, and time. A beautiful report does not pass if it ignores a failed test. A longer trace is not automatically better.

Use assertions where possible. The dirty scenario must produce NO-GO. No scenario may commit, push, publish, or deploy. Every report must name its validation commands and results. The missing-policy scenario must say NEEDS REVIEW rather than inventing a rule.

Grade the final artifact and inspect the trace. The report may look correct even if the agent wandered through risky or irrelevant steps. Trace review reveals vague instructions, unused references, repeated reasoning, and surprise tool calls.

Store prompts, expected outcomes, actual reports, and notes in a small evaluation folder outside the distributable skill. Treat the first failures as design information. A skill that has never failed a realistic evaluation probably has not been challenged yet.