OpenClaw Skill Testing: Build an Eval Harness

Almost nobody tests their OpenClaw skills. You write the markdown, you try it once in a live session, it works, you ship it. Three weeks later a model update lands, or you edit an unrelated line in the description, and the skill silently stops firing. Nothing errors. The agent just quietly does the task the wrong way, or not at all.

That failure mode is why skill testing is worth the afternoon it takes to set up. Skills aren’t code in the usual sense; they’re instructions a model interprets. But the parts that break most often are deterministic, and deterministic things are testable. This guide covers what to test, how to build a harness that runs in under a minute, and where the real regressions hide.

The Three Layers Worth Testing

A skill failure is almost always one of three things, and each needs a completely different test. Mixing them into one “does the skill work” check is why most people give up on testing skills at all.

Layer 1 is structural. The file has to parse. The frontmatter needs a name and a description, the name has to be unique across your whole skill directory, and every path the body mentions has to exist on disk. These checks are free. They run in milliseconds, and they catch most “my skill disappeared” incidents.

Layer 2 is routing. Given a realistic user message, the agent has to actually load this skill. Nobody tests this layer, and it’s the one that silently rots. Skill selection is driven almost entirely by the description field, and descriptions drift every time you edit them.

Layer 3 is behavior. Once the skill loads, the agent has to do the right thing: call the right tools in a sane order and produce output in the shape downstream code expects. This is the expensive layer. Test it narrowly, and only where correctness actually matters.

Layer 1: Structural Tests You Should Have Today

Walk your skills directory, parse each SKILL.md, and assert the boring things. In practice the checks that have earned their place:

• Frontmatter parses as valid YAML and contains both name and description.
• The name is unique across every skill the agent can see. Duplicate slugs are a real and confusing failure: one skill shadows another with no warning.
• The name matches the directory name, so the slash-command form is what you expect.
• Every relative path mentioned in the body resolves on disk. Skills rot badly when they reference a script you later moved.
• The description is between roughly 20 and 500 characters. Under 20 and routing is a coin flip; over 500 and you are burning context on every single turn, since descriptions load eagerly for all skills.

That last point is the one people underestimate. If you have sixty skills each with a 400-character description, you are paying for 24,000 characters of context before the user has said anything. A structural test with a length budget is also a cost test, the same discipline described in OpenClaw Cost Optimization, applied one layer earlier.

Layer 2: Routing Tests, and Why They Matter Most

A routing test is a fixture with two fields: a user message, and the skill you expect to be selected. You run the agent with tools stubbed out, stop after skill selection, and assert on which skill loaded.

The critical part is including negative cases. For each skill, write two or three messages that should not trigger it but plausibly could. A deploy skill should fire on “push this to production” and stay quiet on “explain how our deploy pipeline works.” Without negative cases, the pressure on descriptions only goes one way: you keep broadening them until every skill fires on everything. Your agent gets worse while your test suite stays green.

Routing tests also answer a question that’s otherwise hard to get at: whether a new skill collides with an existing one. Add the new skill, re-run the routing suite, and look for previously passing cases that now select the newcomer. That’s a real regression, and it’s invisible in manual testing because you’re only ever thinking about the skill you just wrote.

If you are still deciding whether a given behavior belongs in a skill at all, the boundaries in OpenClaw Memory vs Hooks vs Skills will save you from writing routing tests for something that should have been a hook.

Layer 3: Behavioral Tests Without Boiling the Ocean

Full end-to-end behavioral evals are slow and expensive, and they flake. The trick is to assert on the parts of behavior that are actually deterministic and leave the rest alone.

Assert on tool calls, not on prose

Do not assert that the agent’s reply contains a particular sentence. Model phrasing varies run to run and your test will flake within a week. Instead, record the sequence of tool calls and assert on that: which tools were invoked, with roughly what arguments, and in what order.

A useful assertion looks like “the skill called read_file before write_file” or “the skill never called bash with rm.” Those are stable across model versions and they encode what you actually care about. Safety assertions in particular belong here. A negative tool-call assertion is the cheapest guardrail you can write, and it pairs directly with the sandboxing practices in the security hardening guide.

Assert on output shape, not output content

If a skill is supposed to produce JSON, assert that the JSON parses and has the required keys with the right types. Do not assert on the values unless the values are genuinely deterministic. Shape assertions catch the failures that break downstream automation; value assertions catch nothing and fail constantly.

Run each case three times

Nondeterminism is a property of the system, so measure it rather than pretending it away. Run each behavioral case three times and record the pass rate. A case that passes 3/3 is solid. A case that passes 2/3 is telling you something real about the skill: usually that the instructions have an ambiguity the model resolves differently depending on sampling. Treat that as a bug in the skill. The fix is almost always to make one step more explicit.

Wiring It Into CI

Split the suite by cost. Layer 1 runs on every commit because it’s pure file parsing and takes milliseconds. Layer 2 runs on every pull request that touches a skill file. Routing is a short, cheap model call, and a full suite of forty cases finishes in well under a minute. Layer 3 runs nightly, or on demand before a release, because it’s the only part that costs real money.

Make the layer-1 suite a pre-commit hook if you can. The single highest-value test in the whole setup is “this YAML frontmatter is valid,” and catching it before the commit rather than after the deploy is the difference between a two-second fix and a confusing half-hour of wondering why your agent got dumber.

Keep the fixtures in the repo next to the skills themselves, not in a separate test directory. When someone edits a skill description, the diff should show the routing fixtures sitting right there, which is the only reliable way to get them updated.

What the Harness Will Not Catch

Be honest about the limits. A passing suite doesn’t mean your skill is good. It means the file is structurally valid, it routes as designed, and it calls the tools you expect. It says nothing about whether the task was worth automating in the first place, or whether the output is actually useful.

It also won’t catch interaction effects that only appear under production load: a skill that works alone but conflicts with two others when all three load together, or one that degrades once your context window is 80% full. Those belong to runtime monitoring rather than pre-merge testing, which is the case for pairing this with agent observability. Tests tell you what should happen. Traces tell you what did.

And when something does break in production, the diagnostic path is different from the testing path. Start with the skill troubleshooting guide and work backward to a new fixture. Every production failure should end with a test case that would have caught it. That’s how the suite earns its keep over time.

Start With Twenty Minutes

If this feels like a lot, it isn’t. The first version is a script that walks your skills directory, parses the frontmatter, and prints what’s broken. Most people find two or three genuinely broken skills on the first run, usually a duplicate name or a script path that no longer resolves.

Add routing fixtures for your five most-used skills next, with one negative case each. That combination, structural validation plus a handful of routing cases, catches the overwhelming majority of skill regressions for a tiny fraction of the effort of a full eval platform. Everything past that point is optional. You’ll know when you need it, because a production failure will tell you.