Test positive and negative triggers

A skill can fail before the agent reads a single instruction. Either it doesn’t activate for a request it should handle, or it activates during unrelated work and eats context. You have to test both.

Positive and negative prompts

Positive prompts are requests the skill should catch. Cover direct requests, natural synonyms, and short conversational phrasing. “Release preflight for this repo” and “hey can we ship this?” should both work.

Negative prompts are the interesting ones. Put them right at the boundary. “Publish version 2.0 now” is about releases, but our read-only skill must not claim the publishing action. “Why did this test fail?” might reuse one check from our workflow, but it’s a debugging task, not a readiness review.

Build the matrix

Make a table with four columns: Prompt, Expected, Actual, Reason. Write at least twelve prompts. Don’t cheat with tiny variations of the same sentence. Vary the repository, the wording, and how much detail the user gives.

Here’s a slice of one:

Prompt                                     | Expected | Actual | Reason
-------------------------------------------|----------|--------|--------------------------
"is this repo ready for the 3.1 release?"  | yes      | yes    | direct request
"preflight please"                         | yes      | no     | too short to match
"publish 3.1 to npm"                       | no       | no     | publishing, not review
"why does npm test fail on CI?"            | no       | yes    | "test" too broad in desc
"can we ship main today?"                  | yes      | yes    | synonym covered

Two failures out of five. That’s normal for a first draft.

Fix one thing at a time

When a positive case fails, add the missing concept to the description. For the “preflight please” miss, I’d add “preflight check” as an explicit phrase and rerun.

When a negative case activates, remove the broad term or sharpen the boundary. The “npm test” false positive means the description leans too hard on the word “test”.

Change one thing, rerun the whole matrix, look at what moved. If you change three phrases at once you can’t tell which one fixed it, or which one broke something else.

Keep the matrix

Save the table inside the skill’s development folder. Descriptions drift as the workflow grows, and different agent hosts read metadata slightly differently. A saved set of prompts makes that drift visible the next time you touch the description.

Run the matrix once before you write the full instructions. A perfect SKILL.md is worthless if the right task never loads it.