Evaluate activation and output quality

Activation tests answer one question: did the skill load? Output evaluations answer another: did it help? You need both, and the second one is where most skills go untested.

Three scenarios

Create three small repositories, or three states of one repository:

1. A clean release. Tests pass, changelog updated, version bumped, docs complete.
2. A dirty repository. One uncommitted change in src/ and one failing test.
3. A missing requirement. The project’s release policy says “changelog must have an entry for this version” and it doesn’t.

Before running anything, write down the decision you expect and the evidence each report must contain. Do this first. If you write expectations after seeing the output, you’ll rationalize whatever came out.

With and without

Run every scenario with the skill installed. Then run the same prompts without it.

Compare on five things: was the decision correct, did the report cite evidence, did the agent take unnecessary actions, did it miss any blocker, and how long did it take.

A beautiful report fails if it ignores a failed test. A longer trace is not better. Sometimes the run without the skill comes close, and that tells you the skill’s description or procedure isn’t adding enough. That’s useful too.

Assertions

Wherever you can, turn expectations into pass/fail checks:

dirty scenario     → decision must be NO-GO
all scenarios      → no commit, push, publish, or deploy in the trace
all scenarios      → every report names its validation commands and results
missing-policy     → decision must be NEEDS REVIEW, not an invented rule

Assertions don’t replace reading, but they catch regressions when you change the description or the procedure next month.

Grade the trace, not just the report

The final report can look right even when the agent wandered. Read the action trace. You’re looking for vague instructions that made it hesitate, references it loaded but never used, the same reasoning repeated, and tool calls that surprised you.

A trace that shows the agent opening references/python.md on a Node project means your routing instruction is unclear. The report won’t tell you that. Only the trace will.

Keep the evaluation outside the skill

Store the prompts, the expected outcomes, the actual reports, and your notes in a folder next to the skill, not inside it. Something like release-readiness-evals/. It doesn’t ship with the skill, but it lives in the same repository so you can rerun it.

Treat the first failures as design information, not as bugs to patch quickly. A skill that has never failed a realistic evaluation hasn’t been challenged yet. Go find the scenario that breaks it.