A skill file is untested code until proven otherwise

A skill file looks like documentation -- markdown, prose, maybe a YAML header -- and that appearance is exactly why it so often ships without the scrutiny a function with the same blast radius would get. But a skill is executable in every sense that matters: it's read by a model and directly shapes what that model does on every request it matches. An untested skill is untested code that happens to be written in English.

This is deliberately scoped narrower than general agent evaluation. Evaluating whether an agent's output is good (see this site's coverage of agent evaluation at scale) is a downstream question about a whole system's behavior. This article is about the upstream question: before a specific skill file is trusted at all, does it actually do what its description claims, does it avoid firing when it shouldn't, and if it claims to self-check its own output, does that self-check actually catch a bad answer?

Advertisement

Golden-example sets: the skill's own unit tests

The direct analogue of a unit test suite for a skill is a set of golden examples: realistic inputs paired with the expected shape of output, not necessarily the exact text (LLM output isn't that deterministic) but the properties a correct response must have. For a skill like this site's code-review skill, a golden example is a diff with a known, deliberately-planted bug, and the assertion isn't "the model says X" but "the model's findings include a finding anchored at the planted bug's line, with a correct failure scenario."

Build the set from two sources deliberately. Positive cases -- inputs the skill should fire on and handle well, covering the range of what "review this diff" can actually mean (a one-line fix, a multi-file refactor, a diff with no real bugs at all, where the correct output is "nothing found"). Negative cases -- inputs that are adjacent to the skill's domain but should NOT trigger it, or should trigger a different, more specific skill instead. A skill validation suite that only tests positive cases will pass while the skill quietly fires on everything.

- id: assignment-in-condition
  input: "diff:\n+ if (user.plan = 'pro') { return 0.2 }"
  expect:
    finding_present: true
    anchors_line: 2
    fix_mentions: "=="
- id: clean-diff-no-issues
  input: "diff:\n+ function add(a, b) { return a + b }"
  expect:
    finding_present: false
    explicit_no_issues_statement: true   # not silence -- a stated verdict
- id: not_a_review_request
  input: "explain what this function does"
  expect:
    skill_should_not_match: true
Advertisement

Regression testing on every edit

A skill file is edited far more casually than a function usually is -- it's prose, so the temptation to tweak a sentence "real quick" without re-running anything is strong, and that's exactly the moment regressions sneak in. Wording changes that seem cosmetic can change model behavior in ways a code diff never would: reordering two instructions, softening an imperative into a suggestion, or trimming an example can measurably shift what the model does on the same input.

The fix is procedural, not clever: the golden-example set from the previous section is not a one-time acceptance test, it's a regression suite that reruns on every edit to the skill file, the same discipline as running a test suite before merging a code change. Track pass rate over time per skill; a skill whose golden-set pass rate drops after an edit that was supposed to be unrelated is telling you the edit had a side effect the author didn't intend -- exactly the signal a code diff's test suite exists to surface.

Because model outputs aren't bit-for-bit deterministic, "pass" has to mean the output satisfies the expected properties (finding present at the right anchor, correct verdict on a clean case) rather than exact string match, and a flaky-looking regression is worth investigating the same way a flaky test in any other system is worth investigating rather than dismissing as noise.