OpenClaw Skill Testing: Build an Eval Harness

Almost nobody tests their OpenClaw skills. You write the markdown, you try it once in a live session, it works, you ship it. Three weeks later a model update lands, or you edit an unrelated line in the description, and the skill silently stops firing. Nothing errors. The agent just quietly does the task the wrong way, or not at all.

That failure mode is why skill testing is worth the afternoon it takes to set up. Skills are not code in the usual sense — they are instructions interpreted by a model — but the parts that break most often are completely deterministic and completely testable. This guide covers what to test, how to build a harness that runs in under a minute, and where the real regressions hide.

The Three Layers Worth Testing

A skill failure is almost always one of three things, and they need completely different tests. Mixing them into one “does the skill work” check is why most people give up on testing skills at all.

Layer 1 is structural. Does the skill file parse? Does the frontmatter have a name and a description? Is the name unique across your whole skill directory? Do the file paths referenced in the body actually exist? These tests are free, they run in milliseconds, and they catch the majority of “my skill disappeared” incidents.

Layer 2 is routing. Given a realistic user message, does the agent actually load this skill? This is the layer nobody tests and the one that silently rots. Skill selection is driven almost entirely by the description field, and descriptions drift as you edit them.

Layer 3 is behavior. Once the skill loads, does the agent do the right thing — call the right tools, in a sane order, and produce output in the shape downstream code expects? This is the expensive layer. Test it narrowly and only where correctness actually matters.

Layer 1: Structural Tests You Should Have Today

Walk your skills directory, parse each SKILL.md, and assert the boring things. In practice the checks that have earned their place:

• Frontmatter parses as valid YAML and contains both name and description.
• The name is unique across every skill the agent can see — duplicate slugs are a real and confusing failure, where one skill shadows another with no warning.
• The name matches the directory name, so the slash-command form is what you expect.
• Every relative path mentioned in the body resolves on disk. Skills rot badly when they reference a script you later moved.
• The description is between roughly 20 and 500 characters. Under 20 and routing is a coin flip; over 500 and you are burning context on every single turn, since descriptions load eagerly for all skills.

That last point is the one people underestimate. If you have sixty skills each with a 400-character description, you are paying for 24,000 characters of context before the user has said anything. A structural test with a length budget is also a cost test — the same discipline described in OpenClaw Cost Optimization, applied one layer earlier.

Layer 2: Routing Tests, and Why They Matter Most

A routing test is a fixture with two fields: a user message, and the skill you expect to be selected. You run the agent with tools stubbed out, you stop after skill selection, and you assert on which skill loaded.

The critical part is including negative cases. For each skill, write two or three messages that should not trigger it but plausibly could. A deploy skill should fire on “push this to production” and not on “explain how our deploy pipeline works.” Without negative cases, the optimization pressure on descriptions is entirely one-directional: you keep broadening them until every skill fires on everything, and your agent gets worse while your test suite stays green.

Routing tests also give you a cheap way to answer a question that is otherwise unanswerable: does this new skill collide with an existing one? Add the new skill, re-run the routing suite, and look for previously-passing cases that now select the newcomer. That is a real regression, and it is invisible in manual testing because you are only ever thinking about the skill you just wrote.

If you are still deciding whether a given behavior belongs in a skill at all, the boundaries in OpenClaw Memory vs Hooks vs Skills will save you from writing routing tests for something that should have been a hook.

Layer 3: Behavioral Tests Without Boiling the Ocean

Full end-to-end behavioral evals are slow, nondeterministic, and expensive. The trick is to assert on the parts of behavior that are actually deterministic, and to stay away from the parts that are not.

Assert on tool calls, not on prose

Do not assert that the agent’s reply contains a particular sentence. Model phrasing varies run to run and your test will flake within a week. Instead, record the sequence of tool calls and assert on that: which tools were invoked, with roughly what arguments, and in what order.

A useful assertion looks like “the skill called read_file before write_file” or “the skill never called bash with rm.” Those are stable across model versions and they encode what you actually care about. Safety assertions in particular belong here — a negative tool-call assertion is the cheapest guardrail you can write, and it pairs directly with the sandboxing practices in the security hardening guide.

Assert on output shape, not output content

If a skill is supposed to produce JSON, assert that the JSON parses and has the required keys with the right types. Do not assert on the values unless the values are genuinely deterministic. Shape assertions catch the failures that break downstream automation; value assertions catch nothing and fail constantly.

Run each case three times

Nondeterminism is a property of the system, so measure it rather than pretending it away. Run each behavioral case three times and record the pass rate. A case that passes 3/3 is solid. A case that passes 2/3 is telling you something real about the skill — usually that the instructions have an ambiguity the model resolves differently depending on sampling. That is a bug in the skill, not a flaky test, and the fix is almost always to make one step in the skill more explicit.

Wiring It Into CI

Split the suite by cost. Layer 1 runs on every commit — it is pure file parsing and takes milliseconds. Layer 2 runs on every pull request that touches a skill file; routing is a short, cheap model call and a full suite of forty cases finishes in well under a minute. Layer 3 runs nightly, or on demand before a release, because it is the only part that costs real money.

Make the layer-1 suite a pre-commit hook if you can. The single highest-value test in the whole setup is “this YAML frontmatter is valid,” and catching it before the commit rather than after the deploy is the difference between a two-second fix and a confusing half-hour of wondering why your agent got dumber.

Keep the fixtures in the repo next to the skills themselves, not in a separate test directory. When someone edits a skill description, the diff should show the routing fixtures sitting right there, which is the only reliable way to get them updated.

What the Harness Will Not Catch

Be honest about the limits. A test suite that passes does not mean your skill is good — it means it is structurally valid, it routes as designed, and it calls the tools you expect. It says nothing about whether the task is worth automating, whether the output is useful, or whether the skill is faster than doing the thing by hand.

It also will not catch interaction effects that only appear under production load: a skill that works alone but conflicts with two others when all three load together, or one that degrades once your context window is 80% full. Those belong to runtime monitoring rather than pre-merge testing, which is the case for pairing this with agent observability. Tests tell you what should happen. Traces tell you what did.

And when something does break in production, the diagnostic path is different from the testing path — start with the skill troubleshooting guide and work backward to a new fixture. Every production failure should end with a test case that would have caught it. That is how the suite earns its keep over time.

Start With Twenty Minutes

If this feels like a lot, it is not. The first version is a script that walks your skills directory, parses the frontmatter, and prints what is broken. Most people find two or three genuinely broken skills on the first run — a duplicate name, a dead script path, a description so long it was eating context on every turn.

Add routing fixtures for your five most-used skills next, with one negative case each. That combination — structural validation plus a handful of routing cases — catches the overwhelming majority of skill regressions for a tiny fraction of the effort of a full eval platform. Everything past that point is optional, and you will know when you need it, because a production failure will tell you.