Tests for agent skills

A simple way to know whether a change to a skill made it better or worse.

Want this to exist? Give it a heart. No account needed.

Posted by Antipaywall

The problem

Tools for testing prompts exist and are mature (see the list below). A skill is a different shape: a folder of instructions and scripts that an agent loads and then acts on over many steps. People write one, try it twice and ship it, with no easy way to know whether the next edit helped.

What to build

A test runner for skills: a folder of cases, what a good result looks like for each, and a report of what passed and what got worse since last time. It may well sit on top of an existing prompt-testing tool.

A strong entry

Runs from the command line with almost no setup. Shows the differences between two runs clearly. Cheap enough to run on every change.

Already out there

Found by a GitHub search when this idea was posted. We have not reviewed them. Builders may fork one of these or start fresh.

Going to build it?

Ideas show what people want. They carry no money and no prize of their own: an entry that builds one competes for the main prizes like any other.

Join as a builder